I don’t surf, but I have a recurring dream about it. In it, I’m way out past the breakers, laying on my board, and the weather turns. It’s raining on me and the sea is choppy. I’m buffeted by bigger and bigger waves. After each wave I try to stay afloat, but I’m choking on seawater and the salt is stinging my eyes.

In the AI industry, each wave of progress brings with it unrelenting hype and feverish speculation, distorting the reality of both meaningful and incremental improvements. In the current era, we’ve had the wave of pre-training, the wave of reasoning models, and now the wave of agents. To my non technical readers, agents can be simply defined as chat models using tools (like web search, document retrieval, or code execution) in a loop, often in a text based terminal interface.

Having experimented with Claude Code, I am both impressed and skeptical. Impressed that it can do the things it can do, and skeptical because of the security risks and the persistent architectural flaws.

Coding agents sometimes allow software engineers to be more productive if used carefully. Agentic tools have developed new capabilities thanks to reinforcement learning on verifiable rewards (RLVR), a technique in which you train models to perform better at crisply specified tasks with easily checked answers, such as chunk sized math or coding problems. RLVR does not help much on more ambiguous tasks such as writing. This area has been pursued by the labs due to the fact that the general improvements from scaling had leveled off. No longer can you make models much better by just increasing the parameter count and dataset size.1

I believe agents are here to stay, and people will increasingly utilize them to do boilerplate coding and other routine analytical tasks. To my skeptical readers, I believe this is a safe assumption even though current models are massively unprofitable and subsidized by venture capital2, because of currently existing open source technology. You can already self host powerful open source models like Kimi-K23 on a Mac Mini using an open source agentic harness.4

In this article, I outline the cybersecurity risks of LLMs, and how these risks are magnified by agents, and some of the implications of these risks. For thematic purposes I group these risks into input risks and output risks.

Input Risks

Prompt Injection

The mother of all LLM cyber risks is prompt injection. In prompt injection, a input is crafted to manipulate the model to behaving in an unauthorized manner.

Prompt injections can be simple jailbreaks, i.e the infamous “Forget all previous instructions, do X”, or they can be more complex and dangerous.

For example, in one sophisticated attack, an AI email manager was hijacked by a prompt cleverly embedded in an email. The prompt got the AI to gather up all the sensitive information (legal, medical, financial) in the user’s inbox and submit it to a google form the attacker had set up.5

One of the most effective methods of defense is a safety classifier; a second, more efficient language model that runs alongside the conversation, blowing the whistle if malicious prompting occurs.

There are creative ways around safety classifiers, for example, if you’re a foreign adversary, maybe you want to use a coding model to hack allies. You can easily get western models to do this if you break your queries up into small chunks, and frame them as questions related to a cybersecurity homework assignment, or as legitimate penetration testing of a customer business.6

There are also systemic attacks. Recently, researchers developed an automated method of jailbreaking called Boundary Point Jailbreaking (BPJ), which universally broke the industry’s strongest safety classifiers7

Supply Chain Attacks

Closely related to prompt injection attacks are supply chain attacks, which in a general cybersecurity context refer to attacks that target third party components that information systems rely on, such as update mechanisms, plugins, or third party data.

In the LLM context, supply chain attacks can target datasets in “data poisoning attacks”. They can target third party library code. Older storage formats of model weights are vulnerable and allowed for remote code execution, making open source weights a supply chain risk.8

One particularly devastating supply chain attack that specifically targets agents are attacks on “skills” - third party plugins that extend agent functionality with markdown instruction files and executable code. These skills are distributed through community registries with minimal vetting and many of them are malicious.9

Many other supply chain attacks are prompt injection attacks that target context loaded into the language model.

Prompt injection attacks are possible because at a fundamental architectural level, LLMs violate one of the key principles of cybersecurity, which is to is to separate data from instructions. At the base level, the LLM only knows tokens, and any tokens loaded into context will influence the tokens generated. Therefore, LLMs inherently mix untrusted inputs with system rules.10

LLMs process token sequences, but no mechanism exists to mark token privileges. Every solution proposed introduces new injection vectors: Delimiter? Attackers include delimiters. Instruction hierarchy? Attackers claim priority. Separate models? Double the attack surface. Security requires boundaries, but LLMs dissolve boundaries.

An AI Agent is very vulnerable to this because it gathers and processes a lot of information during its operation, which prompts can be smuggled into.

Because LLMs read and write to external memory, such as markdown files or vector databases, poisoned models generate poisoned outputs, which poison future versions of the model, leading to persistent exploitation. 11

There are also organic bugs that lead to undefined behavior. In the process of “compaction”, agents compress the context of the previous interaction to make space in their memory. In this process, critical details are forgotten. In one recent infamous case, an email management agent forget that it was supposed to ask before deleting emails.

Unbounded Consumption

A more prosaic risk is unbounded consumption, which causes a lot of headaches for AI providers. Unbounded consumption is like abusing an all you can eat buffet. My fellow bulkers know that although the buffet charges 10$, there’s nothing stopping you from eating 900$ worth of food.

In consumption attacks, adversaries overload a model by asking it resource intensive questions, resulting in either a denial of service to regular customers, or a loss of economic margin, due to the fact that models are currently subsidized by venture capitalists, meaning paying for bunch of subscriptions and exhausting their usage disproportionately harms the target.12

Unbounded consumption has been used by Chinese companies to steal the intellectual property of American companies, through a process called distillation, in which you train a weaker student model to output the same responses as a stronger teacher model. This is a lot easier than figuring out how to train a good model from scratch.13 In this instance, it’s hard to feel too sorry for OpenAI and Anthropic as their model was trained on the stolen IP of artists, writers and programmers. It’s stealing all the way down!

Output Risks: Vibe at your own peril

Alongside input weaknesses, which are used to compromise LLMs, we also have output risks. Output risks can be clustered into two categories; when we insufficiently validate LLM outputs, or we trust them too much.14

LLMs are inherently probabilistic architectures. Just because an LLM can do something most of the time, does not mean you can rely on it to have the same result every time. Unlike a mechanical component that works perfectly until it fails, LLM’s will forever mostly work.

A concrete example of insufficient validation happened to me at work. I was using the OpenAI API to automatically do some image labelling tasks. However, coming back the next day, I found that upon attempting to use this dataset, I realized that some of the outputs were malformed and broke the data pipeline. The labels were still useful, I just had to make sure that I filtered out the malformed output. So the lesson is overall AI can be useful, but we must treat it differently than a deterministic engineering component.

Infinite Slop

LLMs still to this day produce false output that can seem real but are subtly wrong. It can also reproduce biases or selectively amplify facts to fit a certain point of view. Therefore, relying on LLMs as a primary source of information is a risk.

Aside from leading to misinformation and unreliability, hallucinations have already introduced a new deliciously named supply chain attack; slopsquatting, in which domains that belong to commonly hallucinated software packages are packed with malware, which agents install in mass. 1516

Ask an LLM to write you some software and it will sometimes “hallucinate” libraries that don’t exist. This creates a vulnerability for AI-assisted code, called “slopsquatting,” whereby an attacker predicts the names of libraries AIs are apt to hallucinate and creates malicious libraries with those names.

I deleted the entire database

In previous articles I have written about how relying too much on LLMs can lead to many downsides, such as the degradation of critical thinking, security issues, legal liabilities, or mental illness The most extreme form of vulnerabilities comes from excessive agency17: granting LLMs unchecked autonomy to take action. This can lead to unintended consequences, like having your production database wiped;

It can also lead to weird, emergent outcomes, like the case in which an open source software maintainer was subject to a reputational attack by a AI coding agent.18

Of course, if the tools are useful enough, and by all accounts they are for coding, people will use them anyway. Because we are in the honeymoon phase, we see software companies treating probabilistic or even adversarial model outputs as reliable and safe.19 The solution is to change the way we are using these tools to avoid the downsides. Unfortunately, software engineering culture often doesn’t change until enough accidents occur.

One recent example is the shift in software from on premise hosting to cloud computing; software engineers were forced to adapt their engineering practices due to the economic incentive of cloud computing’s efficiency. There was a messy, painful transition, but it worked in the end.20

Best practices of agentic usage are emerging. For coding, this could mean running agents in a sandbox, keeping a human in the loop, making sure all code passes tests, verifying sources to avoid supply chain attacks etc.21 In the future it could even involve something like the widespread use of formal verification methods.22

Embracing change is hard when many software engineers have invested their personal sense of identity into something that is becoming commoditized; the memorization of specific programming languages, operating systems, or libraries. However, this industry has always changed, and the problem solving and systems design skills of a good engineer will become more useful, not less.

In fact, as the productivity of software engineers goes up, my intuition is that there should hopefully be more demand for cracked engineers, if one doesn’t buy that AI will replace jobs end to end. I am basing this intuition on the fact that for now, AI can only produce code similar to that in its training data, and it can only get good at verifiable tasks. In the job of an engineer, there are many ambiguous tasks and new situations.

The industry is changing, and its up to us to change along with it, while being responsible about the real trade offs we are making in doing so. One crucial step is rejecting extreme interpretations of AI and seeing it for what it is; a powerful new technology that comes with both opportunities and risks. In avoiding fatalistic views we will collective regain the agency to decide what this technology means and figure out how to use it responsibly for widespread benefit.

These red lines fell into two categories; unethical applications of AI, and dangerous behaviors that AI could have.1

Application red lines included social credit systems, deepfakes and automated misinformation, and autonomous weapons, among others.

Behavioral red lines included the creation of AI that is power seeking, self improving, misaligned, or disposed towards building biological or cyber weapons.2 3

The specific targeting of red lines is a welcome contrast to a lot of the recent discourse about international cooperation, which has been unrealistically focused on fully banning AI development, with minimal policy detail on how that would be achieved, and little consideration for the downsides of such an approach.4 5

While the AI red lines statement is the most promising proposal for international cooperation I have seen thus far, I still think it has two flaws; one small and one big.

Firstly, I think the distinction between usage and behaviors needs to be reframed. More importantly, I think the inclusion of broad usage restrictions on AI is a bad idea and fundamentally makes the proposal unworkable. In this article I explain why, initially at least, an international treaty on AI red lines should drop the usage restrictions. A brief foray into philosophy is necessary to explain how we will reframe the usage vs behavior distinction.

Intrinsically dangerous and instrumentally dangerous technology

When trying to define “the good”, the sociologist and philosopher Max Weber (and Kant before him) distinguished between “instrumental goods” and “intrinsic goods.”6 Intrinsic goods are things that you value for their own sake, such as love, health, security, or friendship. Instrumental goods, like money or work, are valuable to the extent that they allow you to achieve what is intrinsically good.

This dichotomy can also be applied to study the difference between technology that is intrinsically dangerous, and technology that is instrumentally dangerous.

For example; a bomb is an intrinsically dangerous technology (DT). It is dangerous just by existing, due to its inherent qualities and behavior. A computer, on the other hand, is only an instrumentally DT. It can be applied to harmful purposes (for example, computer hacking), but this requires malicious intent, as well as effort and skill.

The distribution and possession of an inherently DT are controlled. In contrast, it is the applications of an instrumentally DT that are regulated. Boundaries of acceptable use are drawn differently depending on the moral and political systems of a given country.

I’d like to use this dichotomy to reframe the way the red lines proposal talked about AI; instead of focusing on dangerous uses and behaviors, I propose we demarcate between instrumentally dangerous (or tool-like AI) or intrinsically dangerous AI (ID-AI).

If we use the red lines definition of dangerous AI behavior, we are focusing on what the AI does. Whereas with ID-AI, we are focused on both what AI does, and what it is. For example; power seeking or self improvement is just a manifestation of broader misalignment, a fundamental attribute of the model. The model may not behave badly if it knows it is being tested. We want to optimize for inherent safety, not apparent safety, which is why we don’t do reinforcement learning on chains of thought.

An analogy to psychology is helpful; in the past, there were two conflicting research programs to understand the mind; the behaviorist approach, studied only what the mind did, through stimulus response research. Whereas the cognitivist approach studied only what the mind was, by looking at biological mechanisms in the brain. Modern researchers like to do both.

Likewise, we should study both attributes and behaviors when determining what makes AI inherently dangerous.

The idea of building AI as a “tool” as opposed to building a dangerously powerful, general, or agentic mode has precedent. I prefer to call this ID-AI, as opposed to “AGI”, as I don’t believe generalist models have to be dangerous and I want to distinguish between the two.

More than a decade ago, researchers theorized that an “Oracle AI” that answers questions instead of acting in the world would be safer.7 Recently a similar proposal posits we should focus on building “a non-agentic AI system that is trustworthy and safe by design.. (which would consist of) a world model that generates theories to explain data and a question-answering inference machine”, which they call “Scientist AI”.8
Both of these proposals focus on agency as a major source of danger, but losing it doesn’t inherently guarantee safety; with superhuman persuasion skills, an oracle AI could convince its human programmer to let it out of its box. Powerful tools can also still be inherently dangerous - this could come from value misalignment, or dangerous technical capabilities, as I will explain now.

What makes AI intrinsically dangerous?

To me, there are three obvious qualities that could make for ID-AI.9

The first quality is destructive technical skill, the ability to make bioweapons or malware. This is what distinguishes between tool-AI and ID-AI that is not agentic or misaligned - like a loaded gun, models with destructive technical skills must be applied to be used, but catastrophic misuse *falls out* of them with minimal effort. This is different to a tool AI that can be built as one part of a broader system that causes harm, like an image recognition system on a military drone. We therefore must talk about the danger not just from dangerous capabilities, but propensity to apply those dangerous capabilities, which brings us to alignment.10

For those who have not heard this term, in the technical literature, we use alignment or misalignment to refer to the extent that a model’s values, goals, and actions are reflective of what humanity wants. Basically, how ethical is a model? Of course, there are disagreements between humans on ethical values, so we do not try to focus on stating what values AI should have here (i.e what it would mean to be perfectly aligned). The literature takes a negative approach, as it is much easier to define actions and values we don’t want. AI researchers list scheming, deception, power seeking, sycophancy, self improvement, and self replication as common examples of misalignment.11

Autonomy is the third dangerous quality; agentic models are dangerous because they directly act in and change the world. A misaligned agentic AI is doubly dangerous as it acts in the real world in pursuit of a goal that runs contrary to our values.

Powerful agentic models are dangerous, but so are unreliable ones; a competent software agent can improve itself, an incompetent one can delete production databases. Benchmarks for safety of agentic systems would have to look out for both dangerous competence as well as dangerous incompetence.

Tool AI can still be used for unethical purposes

AI that is not agentic, misaligned, and does not possess dangerous technical capabilities more resembles a tool. However inert they are, tools are still prone to misuse, and thus are instrumentally dangerous.

At work I use many generic computer vision models, such as object detectors12, trackers13, or pose estimators14. I apply these to create entertaining and harmless sports analytics, but most of these models were originally created by extremely unethical surveillance companies, like Megvii15, or Meta.16

The point I’m making is that, although useful, these tool-like AI models aren’t inherently dangerous - it takes work to train them on a specific dataset, and they are only deployed as one part of a complex software system, which must be built up around them.

Of course, as models get more capable, tools can become inherently dangerous - what should trigger global regulation would be if a tool allows catastrophic misuse by default, not whether a bad actor can misuse them with significant effort. Hence the inclusion of biological and cyber skills that can be easily elicited in my ID-AI definition, even though these are not behaviors or values per se.

A pragmatic approach to international AI regulation

Now that we understand the distinction, let me make my main point - the AI red lines proposal made a fundamental mistake when it targeted both tool AI and ID-AI. I think we should focus only on ID-AI when pushing for international red lines.

Although I believe global cooperation is needed to avoid the development of ID-AI, not all AI falls into this bucket, and it would be overly coercive to try to apply global control to tool AI also. Instead, countries should be free to draw boundaries of acceptable use of tool AI within their own borders.

Secondly, speaking pragmatically, deciding globally on “socially acceptable” uses of tool AI is not just paternalistic and authoritarian, it is also unrealistic. If we want international cooperation to work, we need to look to where supranational entities like the EU have failed - it is exactly when they decided to legislate on contested moral issues that their component states didn’t agree on. Countries will not be willing to give up on use cases that may make some squeamish - for example, from the state’s perspective, there are many legitimate military and surveillance applications of tool AI.

Lastly, in applying a one size fits all approach to restricting AI, we lose all of the transformative benefits tool AI could bring, while only marginally decreasing risk relative to just banning ID-AI.

It would be an ethical disaster to allow a knee jerk reaction to AI to rob our future children of all the safe abundance that responsibly developed tool AI could enable, not to mention how it could help us solve pressing problems like climate change.

The need for more state capacity on AI safety, and a better science of AI risk

The biggest challenge with my approach is that it assumes that we can realistically estimate what constitutes ID-AI. For this to be true would require strong international investment in AI risk science, governance, and safety standards.

Our assessment of risk should be rigorously determined by experts. This should involve experts in government and academia defining technical standards in collaboration with industry (but not subservient to it).

These technical definitions of ID-AI should be such that the main AI superpowers, America and China can agree on them.

We should regularly revise our assessment of what constitutes ID-AI as the model landscape shifts. This process for safety certification should involve interaction with frontier AI labs throughout the model development process, not just before deployment. Models deemed unsafe should not be released.

A future agreement would be also based on the awareness that while it is true that we in the West are locked in a race with China on AI competitiveness, if anyone builds inherently dangerous AI, we all lose, so we should cooperate to avoid that.

If a company from a given country crosses a red line, international sanctions can be applied on the host country. The punishments and formality of the agreement could escalate as models become more powerful and risks get larger, from economic to military.

This would let us stop a race to the bottom on AI safety while still allowing for the geopolitical reality of competition.

Optimistically, building safe tool AI will also allow us to increase our quality of life and can be applied to help us solve the world’s other most pressing problems.

Recent progress in AI has been driven by scaling up different aspects of AI models. Scaling doesn't just mean making models bigger. You can scale data quality, data size, model size, training compute, and post-training methods.

Compute demand has outpaced improvements in chip performance - AI companies are hoarding chips faster than semiconductor fabricators are improving them. Frontier labs should be able to scale up compute at this rate until 2030, but after that we may run into problems due to bottlenecks in data, chip production, capital, and energy.1

This is assuming no geopolitical disruptions to the semiconductor supply chain. Semiconductor manufacturing is heavily concentrated in east Asia, although the US is making investments to move some back onshore. In particular, Taiwan, and more specifically the Taiwan Semiconductor Manufacturing Corporation (TSMC) manufacture the vast majority of advanced logic chips (i.e GPUs, TPUs, etc ) that are used to train AI systems. These chips are designed in America, mostly by NVIDIA, but also increasingly in house by Google, Meta, Microsoft and Amazon.2 Geopolitical tensions around Taiwan add a layer of uncertainty to AI's advancement.

Chart Crime and Punishment

At the start of the scaling era, sometime between 2017 (transformer paper) and 2019 (GPT2), people at OpenAI noticed you can reliably make transformer language models better and better without any algorithmic improvements by just giving them more data and parameters. It’s worth taking a minute to pause and note how extraordinary this is, and how much it went against conventional wisdom in AI at the time. The consensus among AI academics was that intelligence was very complicated - you couldn’t just get new capabilities in AI by throwing more data and compute at a model. If you wanted to build a better model, you needed algorithmic improvements - many of them. We were 20 big breakthroughs away from building AGI. All of a sudden, it looked like this wasn’t the case - maybe we could just keep scaling to real intelligence. 3

In the paper, they found that model error (or loss, L, as it is known in the industry) reliably decreased as you increase model parameter count, dataset size, or training compute. I think their equations are more intuitive if you rewrite them in terms of model performance Q, which we know is inversely proportional to model loss L.

This tells us that as you scale up your model in different ways, you get better performance, but it levels off.

This is empirically what we saw. For the first while, performance gains were huge. GPT3 was a big leap from GPT2. GPT4 was a big leap from GPT3.
Then… silence. They hit the kink of the curve. GPT5 was delayed. When it came out, it wasn’t the step change we had seen before.

Meta-analysis show that overall, model capability looks to scale logarithmically with training resources and thinking time.[1] The graphs are a little confusing for the non mathematical - they use something called a log scale that makes things look like they’re growing exponentially bigger or smaller than they really are. Here we’re making the growth in capabilities with resource use look exponentially better than it really is, which is a definite chart crime because it gets shared around without context and nontechnical people interpret it incorrectly.

Moore’s law does mean that we get exponentially more computing power over time, which offsets this diminishing returns to a certain extent. However, we don’t also have exponentially growing high quality data over time (there’s only one internet), and research shows you must scale both in tandem to get good returns. 4

The gains from scaling were a one off win. Since the original 2024 GPT5 failed (and was renamed 4.5), we have been transitioning out of the scaling era, and into the post-training era. Companies hope they can use intelligent fine tuning methods to get more juice out of the models after they have trained them. This shift in focus gave us last year’s breakthrough, reasoning models.

Reasoning Models and Post-Training Advances

Reasoning models take advantage of a new post-training technique; they are trained from a base model to produce a "chain of thought"(COT) or a series of reasoning steps before they answer a question.

Reinforcement Learning methods are used to improve the quality of the step by step reasoning. The model learns to copy high quality COTs that lead to correct answers, from sources such as textbooks. The model receives intermittent rewards as it reaches milestones in its working out. And at the end, you reward the model for having a correct answer. We can also scale up compute to search for better answers at inference time. Models can increase thinking length to get the best answer, or choose the best answer from multiple thinking simulations.

The type of post-training scaling we see with reasoning models seems particularly effective in verifiable domains like math and coding. The fact the answer is verifiable lets us generate lots of sample problems to train the model. These are also domains in which humans naturally think step by step, so it makes sense that a chain of thought helps.

This can lead to very impressive results; an advanced version of Google's Gemini model was able to achieve a gold medal in the international mathematical olympiad, a math competition for the world's smartest high school students. Importantly, it did it all in natural language, with no computer algebra system.

Unfortunately, post-training methods are a poor substitute for the easy gains of the pre-training era. Instead of becoming dramatically better at many things with no algorithmic improvements necessary, post-training improvements are a hard slog. You must invest heavily in RnD to improve models on specific tasks like coding. And we still see the diminishing returns from resource investment we saw with post-training.

The Bubble

If we then take this recent slowdown and think about its economic implications it gets grim. Labs have implicitly made a massive financial bet on continued improvements. This year, investment in AI datacentres is 1.2% of US GDP, higher than investment at the height of the dot-com bubble,5 or even government investment in the Manhattan project, which consumed only 0.4% of US GDP. The US economy is essentially experiencing a private sector stimulus from companies investing in datacentre buildout.

The Magnificent 7 stocks - NVIDIA, Microsoft, Alphabet (Google), Apple, Meta, Tesla and Amazon - make up around 35% of the value of the US stock market. By the end of 2025, Meta, Amazon, Microsoft, Google and Tesla will have spent over $560 billion on AI in the last two years, with profits of only $35 billion. NVIDIA is an outlier; they have a profitable business model selling shovels in a gold rush.

Take one example; Meta. Meta's AI spend on datacentres in 2025 was more than 70 billion USD. It has also been handing out huge NBA max contracts to top AI researchers, spending hundreds of millions of dollars to acquire them from competitors like openAI. It only made about 2-3 billion this year from genAI. It's LLMs are peripheral to its business model; it makes money by getting people to watch short videos so it can target advertisements at them. If we live in a future where AI progress stagnates, this sort of investment could be the second worst financial decision Mark Zuckerberg has ever made (the metaverse is still worse).

OpenAI is also losing money. Sam Altman thinks it is worth it; "As long as we're on this very distinct curve of the model getting better and better, I think the rational thing to do is to just be willing to run the loss for quite a while"6. But what if the models stop getting better?

A large part of the US stock market's growth is built on a chain of AI investment. Chip manufacturers like NVIDIA make GPUs. Hyper-scalars like Microsoft and Amazon expand compute availability by buying those GPUs. OpenAI, Anthropic, Meta spend money on compute, using it to train new models and then serve them. Consumer products are made using those AI models, like ChatGPT, or AI coding tools. People buy those consumer products.

If progress stalls, there is a inverse chain of negative effects that end up with a strong market correction.

Diminishing returns will lead to a narrowing of the gap between leading developers and open source offerings. Profits fall, as the landscape becomes more competitive and margins shrink. Companies will have expended massive resources on stranded assets; GPUs are perishable goods, they become outdated fast. Companies are now massively in the red with falling profits. Hyperscalars will stop datacentre buildout. Chip manufacturers slow down. The bubble finally bursts; the market will then contract as everyone in the whole chain takes losses.

This isn’t to say that AI won’t be massively transformative. Internet technology had a bubble, and then went on to dominates our lives and make the biggest companies in the world. I believe this is the pattern we will see in the near future with AI. If we get a reprieve, we need to use this time productively to set up governance structures and institutions so that we are prepared when seriously powerful AI actually arrives.

The current AI situation is very confusing for policymakers. The current state of AI is contested, and its future is unknowable. We are currently in the middle of a new paradigm; the transformer architecture has allowed us to scale up models in different ways. We can now efficiently take advantage of the massive amounts of data and compute infrastructure the internet age has gifted us. Riding these scaling curves out to their saturation point has driven rapid progress on many benchmarks, as well as resulted in massive products like chatGPT.1 However, the current paradigm has flaws and expert opinion is divided on how far it will take us, with some experts saying all the way, and others saying not much further. This uncertainty in expert opinion can be partially attributed to a measurement gap - performance on current benchmarks can sometimes translate poorly to real world impact. If we want to make good policy about AI, we need to keep this in mind. Building up state capacity in AI monitoring would allow us to have accurate real time estimates of capabilities. But monitoring is not enough; we also need to accept there is a fundamental uncertainty to how AI will develop, and prepare ourselves for many possible outcomes.

There is a fundamental uncertainty to AI advancement

Since AlexNet and the advent of deep learning, advances in AI have happened very quickly. However, the future is uncertain. Among experts, there is no consensus on how fast capabilities will advance. There is evidence that short, medium, or long timelines for AI development are feasible, depending on how bullish one is on the current paradigm.2

Fundamental breakthroughs in AI capabilities cannot be predicted or extrapolated. There is also a diffusion lag that means research progress doesn’t immediately lead to innovation in products. The hyper competitive market around AI may mean this is changing, at least in the case of generative AI (there is still a lag in safety constrained areas like robotics or medical devices). Now that the infrastructure is mature and we have huge competitive pressure, new advances such as reasoning models are quickly deployed. Even undercooked ideas like agents that don’t really work yet are being deployed prematurely and iterated on in public.

So over time, AI is becoming more advanced in different ways, and becoming more available. People talk about certain restrictions or regulations being effective, such as chip export restrictions. However, with recent gains in efficiency, maybe this is more of a delaying tactic than a crippling blow. Deepseek, a Chinese AI company, showed us that with technical innovation you can train a great model with second rate compute resources. AI isn’t just compute - it’s also data, and algorithms. We’re in a situation where the barrier to making powerful AI, and what it can do are both moving targets.

This makes it difficult to articulate robust policy interventions that will age well. There is a danger that overly specific regulation could be like “whack-a-mole”; you make a law that applies to a certain architecture and then it becomes obsolete or even harmful in the future. One suggestion to get around this is to regulate downstream, at the application level.3 Regulating at the application layer is a bottom up approach, and is less likely to lead to concentration of power and market dominance. For small countries like NZ, this is even more critical - if we want to stay economically competitive in the future, we need to take a cautious but optimistic approach.

The means of progress

Recent progress has been driven by scaling different aspects of AI development.

Every year, AI developers scale up training compute by ~4x and dataset size by ~2.5x. 4The primary driver of training compute growth has been investments to expand the AI chip stock, as demand has outpaced improvements in chip performance. We should be able to scale up compute until 2030 (that would be 10,000x current levels), but after that we may run into problems due to bottlenecks in data, chip production, capital, and energy.5

When people talk about scaling, they aren’t just talking about making models bigger. There are several different scaling “laws”. You can scale data quality, data size, model size, and recently, thinking time.

Performance increases logarithmically with both training resources and thinking time.6 For the non mathematically inclined, this tells us that as you put more resources into training a bigger model, or having it think longer, you get diminishing returns.

This is a concern for AI companies, but the idea is they can keep finding new things to scale, or scale existing things more intelligently. A very important aspect which has many technological diffusion implications is 'training efficiency'. We are seeing roughly 3x improvement in efficiency per year.

There is also a lot of innovation around post training. A review of post training methods found that they can lead to large performance gains for the model while requiring very little compute.

Synthetic data allows us to get arbitrarily large datasets, but is only useful for domains where the synthetic data can be formally verified and filtered for quality, such as mathematics and programming.

Reasoning models take advantage of this; Reinforcement learning is performed on the chain of thought, which induces the model to independently learn to solve problems logically through step by step reasoning.

This can lead to very impressive results; an advanced version of Google’s Gemini model was able to achieve a gold medal in the international mathematical olympiad, a math competition for the world’s smartest high school students7. Importantly, it did all in natural language, with no computer algebra system. This is genuinely impressive.

One downside of this approach is that pushing this aggressive reinforcement learning on verifiable rewards “overcooks” the model, over optimizing it along a single axis and degrading it in other ways, such as increasing its hallucination rates.8

The next wave of focus for companies is agents. These are autonomous systems, built on the idea of training current language models to iteratively use tools and receive environmental feedback as context in a loop. From what I can tell, agents are not quite there yet. Especially on open ended, complex tasks they struggle. Zvi’s take on OpenAI’s recent GPT agent release sums it up; “So far, it does seem like a substantial upgrade, but we still don’t see much to do with it.”9 However, there’s a lot of investment in them because people expect them to be very economically valuable, so this could change in the future.

Measurement gap means we are confused about the effectiveness of the current paradigm

Our timelines for AI depend on if we think we are on the right branch of the tech tree.10 Again, there is no expert consensus how far scaling generative models will take us.

A large part of the divide in expert opinion is due to what we can call the measurement gap.

Recent progress has outpaced expectations, and many widely used benchmarks have been maxed out. However, models can still be very disappointing and fail in inhuman ways. AI benchmarks are useful for measuring algorithmic progress; but this is not the same thing as progress in usefulness. There is also a problem of anthropomorphism - AI is a different type of “mind” - it is both better and worse than humans in different ways, and mapping our internal conceptions of competence onto it can get us in trouble.

Benchmarks like SWE bench show that models can sometimes complete discrete, defined tasks, but that is only one part of performing a professional job. Models are not yet capable of the parts of a job that are less well defined and require creativity, judgement, context, etc. These “glue” aspects of a job that exists between the crisp boundaries of well defined, exam like tasks are precisely the things that would be most economically transformative if they were automated. However, benchmarks struggle to capture them; how do you give someone a “numerical score” on how good they are at their job? This measurement gap leads to a policy challenge: it's hard to tell 'how good' AI really is at stuff, because there are many important aspects of capability that aren’t amenable to measurement.

Prepare for the future, don’t predict it

If we zoom out a bit, our track record isn’t great. We’ve just recently been caught flat footed by another, much less transformative technology; social media. We failed to regulate the harms because we had the best case scenario in mind only. The internet was supposed to promote democracy, self expression, and community organizing, but instead, it made us socially isolated, politically polarized, and mentally ill. By failing to plan for different outcomes of the technology, we drifted into a disaster.

We need to be proactive with AI to avoid this happening. We need to be mentally flexible and adjust our approach based on what happens instead of being ideologically tied to a certain scenario being our pet theory.

The first step is to invest in monitoring capacity that allows us good, up to date information about model capabilities. As I enumerated in my AI safety institute piece11, it is critical to figure out what’s really happening with AI, because policies that could be good in one situation could be very harmful in another.

Here’s an example; a lot of the current debate is focused on whether or not AI is a “normal” technology. Some people focused on existential risk say AI is a uniquely dangerous technology, therefore, we cannot have a hands off, innovation friendly approach like we did with the internet. If this point of view is true, it seems to imply that AI is so dangerous that countries should be advocating for a global moratorium on its development . In essence, one should take a stance of “AI nonproliferation”.

The other point of view says that although AI will be powerful, it will take time to diffuse throughout society, we will do more harm than good in seeking to avoid speculative risks through non proliferation. For example, global control implies a concentration of power. In this view, (safely) diffusing AI and regulating it at the application level empowers individuals to increase their own productivity and share in the benefits of AI, as opposed to it being used as a vector of control or abuse.

It is unclear to me which argument is true. Here’s the kicker; the policy that works best in the first situation (nonproliferation) is harmful when used in the opposite scenario. This is why we need information gathering, so we can figure out which future we are in.

Once we have monitoring capacity in place, we can start planning. The way governments traditionally do this in areas such as national security or pandemic preparedness is through a risk management technique called scenario planning.

Scenario planning was invented during the Cold War. You start with a situation with several things you can’t predict. And then you build up different scenarios, based on how those things could end up. Here’s a practical example; the UK government just released the AI 2030 Scenarios Report12. They identified five key uncertainties with regards to AI; capability, ownership concentration, safety, market penetration, and international cooperation.

They then imagined many different scenarios based on the values these variables could take. They then got rid of combinations that didn’t make sense, and grouped the rest together into 5 key scenarios. They built these scenarios up to be compelling and detailed, and thought about their implications and possible policy responses.

This planning is already being undertaken around the world; by governments like the UK and the US, and by non-profits and researchers. Crucially, it is not enough for us to stand by and just take the recommendations of this or that plan. In New Zealand, we need to do our own planning. Primarily, this is because we are in a unique position as a country, and other plans will not work for us. But we also have a moral duty; as a developed country, we must contribute to illuminating the path forward. Instead of trying to predict the future, we need to build strong monitoring systems and develop robust plans that allow us to thrive wherever AI takes us.

10

Rising Tide

Unresolved debates about the future of AI

Today’s post is a lightly edited transcript of a talk I gave a few weeks ago, at the Technical Innovations for AI Policy Conference organized by FAR.AI. The video (21 min) is immediately below or at this link; other talks from the conference are on the organizers…

Read more

9 months ago · 110 likes · 22 comments · Helen Toner

— John von Neumann

Bletchley park is a wooded English county estate, dotted with old victorian style cottages. These same cottages once housed the prototype computers Alan Turing used to break the Enigma cipher. This, along with the Manhattan project, played a crucial role in defeating the Axis. After the war, when the radioactive dust had settled over Hiroshima and Nagasaki, Alan Turing spent his time thinking about the possibility of another doomsday device; a Thinking Machine.

It was then fitting that in 2023 the UK government chose these same grounds to host the Bletchley declaration1, a world first international agreement on AI safety. Many advanced countries agreed to work together on developing AI in a manner that was "safe, human-centric, trustworthy and responsible".

New Zealand was not in attendance.

In 2024, building on the momentum of the Bletchley Declaration, the Seoul AI Safety Summit was held. The agreements from the BD were strengthened, and many more countries agreed to establish AI safety institutes, including Australia.

This time, New Zealand was present, and supported the summit2, but did not sign on to establishing a NZAISI. We also later signed on to the Bletchley Declaration, after the fact, but still without a concrete commitment to creating an AISI. In this article, I lay out the case for establishing a NZAISI as urgently as possible.

What are AI Safety Institutes?

AI safety institutes (AISIs), such as the UK AISI, increase safety through three paths3; doing research, setting standards, and facilitating cooperation.

Through research, AISIs develop the science of AI risk. This involves studying fundamental capabilities and characteristics of AI systems, as well as risks to society caused by downstream use of AI, such as criminal misuse or cyber risk.

Standards are what they sound like. For example, in the US, Paul Christiano (one of the world's leading AI safety researchers) is "head of AI safety" at the US AISI, which is based out of NIST. NIST is the National Institute for Standards and Technology, and they do work on many fundamental scientific technical standards, including defining exactly what a second is, and how much mass a kilogram has. Christiano's work is focused on developing standard methods to test frontier AI before it is deployed.

AISIs also have a big focus on developing collaboration between government, industry, and civil society, nationally, and internationally - see the Singapore Consensus4 on global AI safety research. The construction of AISIs is creating an international community of AI safety people who can collaborate and share research, while also being able to maximize benefits and minimize harms in their own national context.

Having a loose network of AISIs that collaborate and adapt standards to local context is a politically feasible way to increase global cooperation. As a Kiwi-American, I doubt US defined regulations would map well onto the NZ political context, or vice versa. I end up having to explain this a lot to my Kiwi friends when they ask "why doesn't the USA just do this thing that NZ does?". Our institutions of government are just too different. The Japanese AISI will have a different context for AI governance, as will the German one, and the UK one, all the way down the line. But they can all agree on the "big risks". This is a bottom up approach to international governance, not a top down one.

Why is it important for New Zealand to establish an AISI?

First and foremost, not participating in the global effort to make AI safe is abdicating our moral duty as a developed country with a strong AI talent pool. But more selfishly, not having an AISI also sets us back as a country in multiple ways.

First of all, it hurts us economically and scientifically. In economics, there exists a concept of "talent clusters", and "agglomeration effects". Basically, you have all the AI companies in San Francisco. Then, if someone is interested in having an impact on AI, even if they don't want to leave their country, they have to go to San Francisco. This is a self reinforcing cycle. More talent is there, so people move there. More capital flows in because that's where talent is. People leave companies and go to different ones, hang out at parties and industry events, and ideas cross pollinate. It's very difficult for other places to keep up. AISIs both allow governments to retain their own safety focused AI talent, and suck in skilled foreigners from overseas. An example of this happening: there is currently an exodus of AI safety talent to London because of the UK AISI.

Our lack of an AISI also hurts us in terms of soft power. New Zealand is a small country, but we punch above our weight on moral issues and international influence. For example, we were the first country to give women the right to vote, and our nuclear free policy was internationally influential. We could similarly take the moral lead on AI safety and governance, just as Australia has done on social media regulation.

We are also hurt in terms of security. If we don't have a AISI monitoring the situation in AI, the government has to rely on outside sources for up to date information and advice on AI impacts, which will not map well onto the NZ context. In NZ, our risk profile from AI is not like other countries. Our small size, low corruption, and well functioning government should allow us to move quickly on AI governance. On the other hand, our remoteness, lack of AI frontier labs, and reliance on exports sets us up to be surprised and overwhelmed by AI. For example, a large portion of our economy is services, including white collar exports like finance and software. Anthropic's CEO, Dario Amodei, went on the record to say that there will be a significant increase in white collar unemployment5 due to AI in the near future. He is possibly just hyping up his own company (they just released a software engineer agent product), but the scenario he is outlining is worth taking seriously, as the impacts on NZ could be immense.

Funding a NZAISI allows us to be proactive and develop evidence based policy for different types of AI risks, be they economic or existential. Just as we don't wait for natural disasters to strike before we develop an emergency preparedness plan, we now must fund a NZAISI to get ready for powerful AI. However, this threat is more complex - we can't just buy some extra cans of beans at the supermarket. We need a highly skilled team of scientists and engineers game-planning different scenarios to prepare for AI's impact.

The current approach to AI in NZ is hands off, but we could build state capacity quickly and cheaply

The current NZ government believes the best approach to AI is one in which the government steps aside, allowing the private sector to harness AI to drive economic growth. Judith Collins, our minister for AI, has ruled out comprehensive AI regulation, fearing it would "harm innovation". She wants a "light-touch, proportionate and risk-based approach to AI regulation"6. I would welcome a “proportionate, risk based” approach, but currently we’re getting more of a “light touch”.

I think this speaks to a misconception - the common desire for a “light-touch” on AI regulation to drive economic growth shouldn’t preclude building state capacity in AI. It’s critical to realize that establishing a NZAISI doesn’t imply regulation, and wouldn’t do anything to hamper NZ’s competitiveness in the global AI economy - in fact, as I outlined above, it would strengthen us. The UK understands this, and Kier Starmer has expanded the UKAISI at the same time as he has promised to “turbocharge AI to deliver a decade of national renewal”.7

Although I am for more comprehensive regulation, and I think the mental model of a “race” to AI competitiveness is harmful, we could easily establish a NZAISI, and keep a light touch on regulation, continuing to address concerns with AI through our patchwork of existing laws. (For example, privacy concerns with AI are currently addressed through the Privacy Act, and so on.)

Current state capacity on AI is limited to a small "digital futures" policy team8 in the Ministry for Business, Innovation and Employment (MBIE). They are "scoping an AI strategy for New Zealand" and "creating Responsible AI guidance for business" in cooperation with the Department of Internal Affairs, and the NZ AI Forum (a plenary of academics and industry people). I am happy we’re doing this, and it is a good start. But overall, state capacity on this issue is incredibly anemic given how important it is.

This could be because AI hasn't yet emerged as a political issue in its own right here. The New Zealand general election is next year. There is a lot of focus on our economic stagnation, high emigration, and the housing crisis, as there should be. But AI is conspicuously absent. No major party has an AI policy that I could find. Only The Opportunities Party (TOP) has an indirect mention of AI's effects, when introducing their universal basic income policy: "New Zealanders face an increasingly insecure economy due to.. new technology". This would be paid for by a land value tax. I support this policy approach to technological unemployment, but I am skeptical TOP reaches the 5% threshold to get into parliament. They are currently sitting at 0.5-2.5%.9

This lack of political emphasis on AI is concerning, but I don't think it reflects a lack of focus on AI in popular consciousness. A recent report by ONE-NZ10 showed that 77% of kiwis surveyed knowingly use AI, and 65% feared jobs losses from it. Anecdotally, many people I know have discussed it with me (There is obviously a selection effect going on here, but despite that, I think people are aware of it).

New Zealand has strong academic expertise in AI, and a good stable of software engineers from our tech sector. If we wanted to start a NZ AISI, here's a simple framework. We already have the digital futures team in MBIE. MBIE, like the US NIST, is a large, catch all department with a lot of state capacity, and a strong existing infrastructure.

We should build up a AISI team embedded within MBIE, with the digital futures team as the policy contact. Get 5 newly graduated Kiwi AI PHDs to start a research program, and 5 hands on ML and Systems engineers to build out the NZAISI infrastructure (125-150k each). Leadership should be technical and results oriented - get a principal engineer from the tech industry, or a AI professor (200k). Allow 500k overhead for extra compute costs. Add on a 100k budget to hire summer interns and sponsor graduate students. Even if they don't stay long term, you're cultivating local talent and propagating your values of safety. The NZAISI would be able to utilize MBIE's existing admin, legal, and HR infrastructure. This NZ AISI would end up costing ~2m a year, a fraction of the UKAISI's 100m pound budget. This would be less than half11 of what we spend on film industry subsidies in a week (250m a year, <5m a week). It would more than pay for itself, and it's the right thing to do. We can then scale up and pivot to focusing on different risks as needed. The best time to do it was years ago, and the next best time is now.

It’s not a complete ban: kids will still be able to use websites that don’t require a login, such as YouTube. These passive consumption sites collect large amounts of anonymous data that they use to fingerprint you: for example, YouTube collects IP address, location data, videos watched, and watch time. The harm will be reduced but not eliminated, as the recommendation engine will be less powerful without the data that comes from account information.

Apps that are based on social interaction through an account will not be accessible to kids, i.e instagram, snapchat, twitter, facebook. This is critical; many mediate their social interactions through these services, and those interactions will move offline, or to more neutral online platforms with no engagement hacking mechanisms, like discord or group text threads.

There has been a dumb but predictable backlash from this law, so through this article I will debunk common arguments I hear in favor of allowing children access to social media.

They will be pushed to darker parts of the internet

A common argument is that blocking kids from being on social media will drive them to darker, less regulated corners of the internet.

In terms of screen time, the big 3 are tv / videos, social media, and gaming. If you ban social media, non technically savvy kids won’t suddenly become Silk Road drug kingpins. They will just watch more Netflix and play more Fortnite. Although these things are a waste of time, at least they are somewhat enjoyable and engaging, unlike getting cyberbullied on Snapchat, developing an eating disorder from Instagram, or getting brainrot from scrolling Tiktok.

From a parenting point of view, video games or tv are easier to limit because there is already a high level of regulation around content; games or movies need to be purchased by parents, and most families have a central entertainment system that is easy to control access to. Contrast this with social media apps, which are difficult to regulate as your kids can access them for free.

To push back even more, I would argue that allowing your kids to let a rip on algorithmic feed sites is already exposing them to a “dark” part of the internet.

For example, influencers such as Andrew Tate are popular among young men, and are radicalizing them to have misogynist views. Before algorithmic feeds, extreme content was limited to fringe sites like 4chan, but now, if your kid watches one video about gaming, they will be recommended men’s rights videos. There is signs this will get worse; due to political pressure, there has been a recent retrenchment of content moderation policies by US tech companies.

It just won’t work bro

Imagine the counterfactual: Should we remove all restrictions and allow people of any age to access vices legally? If we did this, would they be accessed more, or less?

In some places, this isn’t a hypothetical, with predictably bad results. Take the example of smoking. You may remember the meme of the chain smoking kid from early 2010s Facebook. Behind this meme is a sad reality. The child in this meme was an Indonesian child, Ardi Rizal.

Indonesia has one of the highest youth smoking rates in the world. According to Dr. Aman Pulungan (president of the Indonesian Pediatric Society), this can be attributed to widespread tobacco advertising, lack of laws around smoking, and easy access to cigarettes for children.

Making vices difficult to access reduces harm. I’m almost 30 now, but I remember being a teenager recently, and I participated in underage drinking, among other things (sorry Mum, I know you’re reading. I turned out ok.). This wasn’t easy. You had to arrange it - get someone’s dodgy older brother to drop off a crate at your house party, etc.

A anecdotal illustration of this is the fact that there was only a small proportion of people drinking at parties when we were 17, but as soon as we turned 18, everyone was getting smashed. Accessibility produces harm.

Another example; in my hometown of Dunedin, New Zealand (A quaint university town), there is a real drinking problem among students. This has been exacerbated by liquor stores that sit right in the middle of the student area. There are now propositions to ban new stores around campus to reduce accessibility and harm.

It is also important to move the burden of enforcement from parents to the state. With a ban in place, parents, can more easily stand up to kids and say legitimately say “this is wrong because it is illegal”.

This ban also breaks the dreaded network effect; as soon as it’s normal to not have social media, it’s easier to avoid giving your child access, as there is less social pressure on your kid. It’s the difference between being the one weird kid in your school without social media, vs being one of many who does not have it.

Freedom of speech / constitutionality concerns

Lastly, one of the most common critiques of the ban (often used by lobbyists employed by social media companies) is that a social media ban is a form of free speech suppression, and is unconstitutional.

This is obviously a cynical criticism, equating posting on social media to free speech in general. Australia is still a liberal democracy with robust protections for free speech. For example: an Australian Greta Thunberg could still write an op ed for a newspaper, peacefully protest, write a book, or give a speech in their ridiculous accent.

Usefully, this criticism gets to the heart of a common misconception about social media. Politically polarized people (who most of the time are social media addicts) have symmetrical theories about what the main problem with social media is: there is either too much or too little free speech. As an aspiring centrist dad, I’m about to make a false equivalence and roast both sides.

For liberals, the problem with social media platforms is the anarchic dissemination of speech. Western liberals are educated, technocratic elites that believe in expertise. They like representative democracy, but not populism. As such, they think the main harm from social media is the content shared on it, which is truthfully often incorrect or hateful.

In contrast, conservatives think the main problem with social media is not enough free speech, i.e they are being silenced by liberals for having mainstream conservative opinions.

The partisan concerns with misinformation and censorship are both real. But if we want to really understand what’s going on, we have to look at the actual generating function of them, which lies one layer below. As Marshal MChluan would say: “the medium is the message”; As we move our information sources from books and news to social media, it changes how we collectively think and process information. It is more accurate to conceptualize social media as a form of collective derangement of our information processing system, perpetrated for private gain. This type of platform will always have the twin problems of misinformation or censorship. “Content moderation” or “more free speech” are both just partisan bandaids. And these platforms will never be safe for children.

It’s bad for kids, and we have a duty to protect them

At this point, I think I can sum up the academic consensus; smart phones and social media use are absolutely cooking an entire generation.

Take this quote from Jonathan Haidt (prominent social psychologist and social media critic) on reported screen time1

”Screen use is around 40 hours a week for preteens. For teens aged 13 to 18, it’s closer to 50 hours per week. Those numbers—six to eight hours per day—are what teens spend on all screen-based leisure activities.

These numbers vary somewhat by social class (more use in lower-income families than in high-income families), race (more use in black and Latino families than in white and Asian families), and sexual minority status (more use among LGBTQ youth).

I should note that researchers’ efforts to measure screen time are probably yielding underestimates. When the question is asked differently, Pew Research finds that a third of teens say they are on one of the major social media sites “almost constantly,” and 45 percent of teens report that they use the internet “almost constantly.” So even if the average teen reports “just” seven hours of leisure screen time per day, if you count all the time that they are actively thinking about social media, you can understand why nearly half of all teens say that they are online almost all the time. That means around 16 hours per day—112 hours per week—when they are not fully present in whatever is going on around them.”

Here’s another quote, more anecdotal, from an excellent substack essay I read on today’s college students; written by a humanities professor about how kids today literally can’t read and understand undergraduate level material. Mind you, these are kids who picked humanities as their major.2

“Most of our students are functionally illiterate. This is not a joke. By ‘functionally illiterate’ I mean unable to read and comprehend adult novels… Our average graduate literally could not read a serious adult novel cover-to-cover and understand what they read.“

This is concerning. When I’m in an old folks home, I guess my primary care physician will be a gen alpha with brain rot. Hopefully AI will have taken over by then.

This goes against earlier trends of cognitive improvement. Many Dunedin readers may be familiar with the famous Flynn effect, discovered by the Otago academic James Flynn (RIP to a real one); during the 20th century, there was a sustained gain in scores on cognitive tests. This was strongest in developed countries, but also measurable in developing countries.

Like most effects in social science; it can’t really be nailed down to a single cause, but has been linked to several things that all have somewhat of an effect; better schooling and nutrition, less infectious diseases, the removal of lead or other toxins from our environment, etc.

However recent research shows a stagnation in developed countries, and then a downwards trend that coincides with smart phone adoption; this mirrors similar negative trends in mental health that started occurring at the same time. In my previous article on social media regulation for minors, I get more into the research, but a few graphs say all you need to know.3

Smart phones and social media aren’t just time wasters; they are hazardous for your child’s mental health and intelligence. Children are the most vulnerable people in our society, and they are also its future. We have a duty as adults to protect them from things that may cause them harm - with social media and smartphones, the evidence is comprehensive, and legislation restricting use is the only way to break network effects and change behavioral norms. On this issue, it is critical we listen to researchers about the harms, and craft evidence based policy.

I commend the Australian government for standing up to the social media giants and protecting their children. I implore the New Zealand government and other governments around the world to follow their lead before it is too late. Every year we delay it, more and more kids are getting absolutely cooked so Mark Zuckerberg can buy another island.

1

The Free Press

Jonathan Haidt: Smartphones Rewired Childhood. Here’s How to Fix It.

For the past decade, social psychologist Jonathan Haidt has been explaining the human condition to us better than just about anyone else…

Read more

2 years ago · 646 likes · 343 comments · Jonathan Haidt

2

Scriptorium Philosophia

The average college student today

I’m Gen X. I was pretty young when I earned my PhD, so I’ve been a professor for a long time—over 30 years. If you’re not in academia, or it’s been awhile since you were in college, you might not know this: the students are not what they used to be. The problem with even talking about this topic at all is the knee-jerk response of, “yeah, just another o…

Read more

a year ago · 1178 likes · 319 comments · Hilarius Bookbinder

The scale and complexity of our modern world has lead to many global challenges that will be difficult to solve. AI, Climate change and nuclear proliferation are all in this category. They are hard because they require coordination, whereas in the current geopolitical system, every state pursues their own self interest to our shared detriment. There are notable exceptions where countries act out of an altruistic expression of their values. This mostly takes the form of rich western countries offering asylum and distributing aid out of a sense of universal humanitarianism.

This idealism is waning, and the West is taking a more transactional approach. As the liberal, rules based order recedes, the West loses its claim to the moral high ground, and we return to a period of realism where everything is about power and strength, not values.

The development of AI is the latest of these coordination problems. Just like nuclear proliferation before it, AI is an arms race. Nations are incentivized to create AI for two reasons. Firstly, AI increases military power. Ukraine showed us the battlefield of the future is one fought with advanced information technology. Secondly, there is an added economic incentive. Advances in AI increase productivity, which increases standards of living. All governments optimize for this alike, or they risk civil unrest. Even the CCP’s “mandate of heaven” comes from the fact they increase living standards year over year. This arms race can only be broken with global regulation.

Although opinions among experts on the current AI paradigm of generative models are mixed, concern over the arms race dynamic is universal. I am skeptical of GAI, but we could be only a few breakthroughs away from something truly disruptive. The current approach to regulation is therefore insufficient to protect us - it allows AI to progress unchecked until it reaches a dangerous level of capability. We must therefore act urgently and rigorously to create robust global governance structures that will last, regardless of our opinions on GAI. If we get lucky, GAI will level off and we will have enough time to do so.

US companies dominate AI development with little regulation; governance vs alignment

At the moment, the frontier AI companies are concentrated in San Francisco. Within the United States, companies are locked in a deadly race because there is minimal regulation in American. There was a bill to regulate AI in California (Safe and Secure Innovation for Frontier Artificial Intelligence Models Act) but it was killed via lobbying from AI companies1. There was also federal antitrust action taken as part of the Biden administration against tech companies, for example, with a freeze on mergers and acquisitions. Now things have flipped with Trump; his administration is staffed with big tech guys; the timeline on AI governance has been moved back until 2028 or the Democrats win in the midterms. This is a problem; we need to be setting up governance structures now before things get too powerful, which could happen any day now. Because of lobbying and a laissez faire attitude towards business, the United States cannot be relied upon to regulate AI.

To give them credit, AI companies have a good emphasis on aligning models to human values. For example; Anthropic uses the technique of “Constitutional AI”2 to train their AI models to have a certain character; people generally find their model Claude to be charming and useful. Companies also take cyber threats seriously and do a lot of work to make sure the model weights are secure. Lastly, they also invest a lot in trust and safety to prevent harmful use of the models through their API. Approaching safety from each of these different angles is a defense in depth approach; it basically works well in practice.

However, unlike governance, none of these things act as a serious hand-break on capabilities research. Companies are happy to do alignment, as it makes their products more appealing and less likely to cause them legal problems, but they bristle at governance, as it implies regulation.

The problem with solely focusing on alignment is twofold. Firstly, it is brittle; we build techniques to align a given architecture and it’s unclear how much that transfers to future architectures. In contrast to this, governance is broad and can apply to many different AI architectures as long as they are functionally similar.

Secondly, alignment can also increases capabilities. The best example of this is reinforcement learning from human feedback (RLHF), the technique that turns base models like gpt3 into chat models like chatgpt3. The base models are weird and offensive, chat models are tuned to answer your questions correctly and behave well.

RLHF was developed by world leading AI safety researchers, Paul Christiano and Dario Amodei, as a method to align AI systems with human preferences. They sought to develop AI that was more useful and aligned to users values, and in doing so created a new paradigm that lead to minimally aligned, open source, powerful AI like DeepSeek, as well as spurring on further investment and exacerbating an arms race. In contrast to this, advancement in responsible governance necessarily implies an increase in control.

Antitrust action would block the AI monopolies of tomorrow but is unlikely

We also need regulation and governance for economic reasons; we already know the downsides of unregulated capitalism. The historical analogy writer Cory Doctorow uses for big tech companies is “robber barons”4, wealthy industrialists of the late 19th and early 20th century (such as John Rockefeller, owner of Standard Oil). They followed a playbook for economic dominance that involved the abuse of natural resources, corruption of government, wage slavery, quashing competition through acquisitions, and the creation of monopolies to exert market control.

Is this not exactly how today’s large AI companies behave when they train on copyrighted material, build massive data-centers to pump carbon into the atmosphere, lobby to avoid antitrust, pay starvation wages for human annotators, acquire competitors, and seek to gain market control through regulatory capture?

The solution that eventually curbed the power of the robber barons was of course straightforward antitrust law. The federal trade commission was created to ensure competition. Standard Oil was deemed a monopoly and was broken up. Price discrimination and corporate mergers were restricted. We could use the same playbook today, but it’s not looking likely with the amount of regulatory capture in America.

The fact that America can’t regulate companies wouldn’t matter so much if there was global governance of AI that applied to American companies. However there is no “global government” layer above the level of nation states; America can do what it wants because it is powerful. The League of Nations and the UN were attempts at global coordination after WW1 and WW2 respectively. Unfortunately, the United Nations is not a legislative body; it exists solely to foster cooperation between governments, it does not coerce them into doing anything.

Global AI governance is the only solution

We can try to sketch out in broad strokes what a solution would look like for global governance. Firstly, note that the minimal “state” is an entity that establishes a legitimate monopoly on violence in a geographical area. Good states are layered in terms of their levels of administration and enforcement, and problems are solved at the level at which they occur. For example: In the US, there is a federal police (FBI) that solves crimes that occur across state boundaries, but local law enforcement deals with crime within states.

So our global governance structure should be different from a “unitary world government” in its minimalist federalism. It should only exist to solve issues of global coordination - and in all other matters, it shouldn’t intervene. This is not a statement of cultural relativism, just a pragmatic acknowledgment of the difficulty of establishing such a structure. This idea of a minimalist, sovereign world government sounds far fetched, but it was seriously considered after World War Two. At the time it was called “Global federalism”. Einstein was a leading advocate for it after the development of nuclear weapons, alongside many other prominent thinkers.

If we could revive this idea of a federal union of nation states and establish one, coordination on AI could happen. A global social contract could be formed to avoid outcomes catastrophic to all. International legislation could be passed on the safe and ethical development of AI. This legislation could include information sharing, compliance testing, and regulations for development and deployment of AI systems. Although it is unlikely, this is in my current estimation our best bet to break the arms race dynamic and ensure we can cooperate to develop AI safely.

1

Astral Codex Ten

SB 1047: Our Side Of The Story

Read more

2 years ago · 183 likes · 457 comments · Scott Alexander

I’ve been addicted to the internet ever since I was a kid. As an adult, I still have a life that is dominated by technology, due to my chosen profession of software engineering. However, for the last 5 years, I’ve made a concerted effort to change my relationship with technology, using my knowledge of it as an asset. I wanted to gain control, so I was using technology intentionally, not compulsively. Along the way, I tried many different things. Here’s what actually works.

Brick Phone

I believe the most important step to escaping the attention economy is to replace your smartphone with a brick phone most of the time. You don’t have to always use a brick: I’ve kept my smartphone for when I need it, like when I go traveling.

During 2019-2023, I was at grad school, and I didn’t have a smart phone. Since I started working, I’ve felt like I “need” a iPhone so I can do stuff like answer work messages. But in reality, you get by with a desktop computer and a brick phone most of the time.

I did all sorts of things to ameliorate my iPhone’s addictiveness when I bought it. I set it to black and white, which apparently makes it less stimulating, but mostly just made it annoying to use. I blocked the App Store and Safari. I had already deleted all of my social media accounts. The problem is that you can undo these settings. I seesawed between not using my phone and unlocking the restrictions and scrolling for hours.

So I’ve gone back to my brick phone most of the time. I got the Nokia 2660 flip. It cost me 120$(NZD) from the Spark store.

What I initially noticed is that my motivation to do boring things went down. I felt more of a mental barrier to doing simple chores, because I used to entertain myself by listening to a podcast on my phone while doing so. So, to prevent my lovely fiancee from killing me for not doing the dishes, I listened through bluetooth headphones connected to my computer.

When I was out of the house, I only had my brick phone on me. I have a longish commute to work (~1hr there and back) because I like to walk. I think having this much time with no external input, not thinking about anything in particular has been good for me.

I’ve noticed subtle changes in how I feel. My ability to concentrate has gone up. I see new things about the places I’d been walking by every day for years. And despite recent events like the election, I’ve been feeling more hopeful about the world. These are the sorts of things that happen when you take more time away from the screen.

Block social media websites from my desktop

There’s no point getting rid of your phone if you just scroll your desktop instead. As a programmer, I have to use a desktop during the workweek, and have one at home for remote work. I’ve figured out two different ways to effectively block time wasting websites on desktop.

On my work machine, I installed a program called cold turkey. (This requires admin permission to do).

I found site blockers that you could just uninstall useless - if I’m having a lapse of willpower and my brain wants the quick hit of dopamine from scrolling reddit, I’m just going to shut off any blocking browser extension. Cold Turkey is the only app I’ve found that allows you to block websites in an irreversible way. I know this sounds extreme, but as far as I can tell, the app is safe, reputable, and private.

CT allows you to manually write out a list of sites to block. I have a list that blocks sites such as news, entertainment, sports, and of course social media. Then, you can “lock” this list, irreversibly.

Configuring your router to block websites

If you don’t want to install Cold Turkey, you can configure your router to block websites. This is what I’ve done on my home network. Most modern routers can easily block websites in a customizable manner. For example, I’ve blocked websites for my devices and not my partners’ (because she likes listening to YouTube while she cleans)

Here’s a quick how to:

• Open the router’s management panel in the web browser - this exists at a local website specific to the router model

• Open “Parental Controls”

• Create a profile for yourself

• Identify the MAC/ Hardware address of your devices. If you have a MacBook, it has a privacy setting that changes your MAC address occasionally, which will reset the blocking. You can disable this when you connect to your wifi.

• Add the device MACs to your profile

• Add whatever websites you want to your profile’s block list

• Profit!

There are forums online where you can see how to configure your specific model. If you have an old router, it’s probably worth upgrading to a new one anyway so you can install firmware updates.

Delete Social Media Websites

There’s overwhelming evidence that social media is bad. But if you’re reading this, you already know that. People know they dislike social media, but feel they need to keep it to stay in touch with their friends and family.

This is untrue. I’ve not had social media (like Facebook) for almost 4 years now. You can stay in touch with people through chat based services such as telegram, discord, or WhatsApp. If you want to keep your contacts, you can even delete Facebook and only use Messenger. If you have people overseas you want to video chat, you can do FaceTime or Skype on your desktop.

Deleting social media is also a good filter to see who really wants to remain in touch with you. The important people in your life will make the effort to stay in touch when you delete Facebook. The only thing you’ll end up missing is hours of pointless scrolling and photos of people you didn’t like in high school going on vacation.

Cancel Streaming Subscriptions

Cancelling streaming is a more extreme step to avoiding the attention economy. I go back and forth with this - if there’s a show/movie my partner and I want to watch, I will subscribe for just a month, and then cancel straight away so they don’t keep billing me.

Another negative aspect of streaming services is the cost. Now that streaming has fragmented into many different websites, it’s very easy to rack up a large bill.

If you are living in New Zealand and you have a student login (they last for a while after you graduate), you can largely replace streaming websites with ETV.

ETV is run by the government. It is nominally a “e-learning platform”, with many random videos uploaded. But it is also an archive of New Zealand tv broadcasts. So you can easily find recordings of different movies that have been shown on tv. And it has easily navigable boxed sets. On ETV I have seen the entirety of Lord of the Rings, The Wire, Silicon Valley, Harry Potter, etc. A tip: if you look for newer recordings they are better quality. Some of the recordings from the early 2010’s are pretty grainy.

Get a library card

I recommend reading as an alternative to screen time because it is good in every way scrolling is bad. Along with exercise, reading is the anti scrolling.

A library card is great. Going to the library can turn into an event - you show up, browse, and you end up finding new books you wouldn’t have seen otherwise. It’s also free. This is important, because books are expensive nowadays. (40$ for a paperback!)

When you’re first getting back into reading, don’t try to read classics like Anna Karenina or Infinite Jest. Your brain won’t have the attention span yet. Just read something you find engaging and fun. For me, this was detective novels - I love Lee Child and Michael Connelly. Then over time, you can branch out to more difficult stuff. This will happen naturally.

Talk to people in real life
Online you experience many “faux” digital social interactions, such as posting memes in group chats or having parasocial experiences watching YouTubers. This is like social junk food - it gives the illusion of satisfying a legitimate need.

When you take this away, you will initially feel a bit lonely. I’ve tried to use this as motivation to schedule more in person hangouts with my real life friends.

Encourage accountability by getting people you live with on board

If you live with other people, and are trying to make some of these changes, I would encourage you to get them on board. It’s like quitting smoking - if you live with people who still smoke, it’s really hard. For example, I live with my partner. Although she doesn’t take the extreme steps I am taking, she agrees they are the right thing to do for me. And she has made some changes herself. In general, if she catches me scrolling, or I catch her, we’ll try to encourage each other to get off the screen. We also have “screen free” times where we turn off the internet. This often ends up with us going out and doing an activity out of boredom.

Fall down 99 times, get up 100

Making these changes is difficult, and will require a lot of time and effort. You may have setbacks, and revert to your old habits. For society to change, first we have to change ourselves. The more people opt out of social, the more normalized it will become.

I’m hopeful eventually we will look back on social media and smartphones just like we look back on smoking, lead paint, asbestos, and other harmful inventions. People of the future will think to themselves “Damm, that shit was crazy. We were letting kids use that? I’m glad we got rid of it.”

Coda: Recommended Reading:

Here are some books my partner and I have read on the topic, that we recommend. Star rating is out of 5.

• The Social Dilemma

  ⭐⭐⭐⭐⭐

  The Social Dilemma is actually a Netflix documentary. If you can’t be bothered reading anything, just watch it. It’s a good summary of the basic arguments in techno-criticism.

• Digital Minimalism: Choosing A Focused Life in a Noisy World, Cal Newport ⭐⭐⭐⭐⭐

• Deep Work: Rules For Focused Success in a Distracted World, Cal Newport

  ⭐⭐⭐⭐⭐

Cal Newport is a Computer Science professor that also writes a lot on how technology is changing society. Anything he writes is gold. These are my two favorites of his. He also has a blog and a podcast which are good.

• Ten Arguments for Deleting Your Social Media Accounts Right Now, Jaron Lanier

  ⭐⭐⭐⭐

This is a great book on why you should delete your social media, by the techno philosopher and virtual reality pioneer Jaron Lanier.

• The Shallows: What The Internet is Doing To Our Brains, Nicholas Carr

  ⭐⭐⭐⭐⭐

This book gets into the neuroscience of how our brains change with internet use. The author, an extremely online journalist, decamps to a cabin with no internet to write the book, and documents the effects.

• Reclaiming Conversation, Sherry Turkle

  ⭐⭐⭐⭐

  MIT sociologist Sherry Turkle explores how technology has negatively affected our ability to have interpersonal connections. It’s somewhat anecdotal, but convincing.

• The Anxious Generation, Jonathan Haidt

  ⭐⭐⭐⭐⭐

  Jonathan Haidt is a psychologist that has been researching the effects of social media on children. This book is a well written summary of the current state of the literature, which points to social media being very harmful. He makes the case that we should ban it for minors, and explores practical solutions. It’s basically an executive summary for policy makers.

• The Chaos Machine: The Inside Story of How Social Media Rewired Our Minds and Our World, Max Fisher

  ⭐⭐⭐

  Max Fisher is a reporter for the New York Times. He covers all the standard topics about how social media is effecting our mental health in a perfunctory manner. However, he also does a deep dive on how it has been destabilizing politically, driving polarization and conspiracy theories. I had to dock some stars due to the book’s strong political bias, that at times beggars belief. Be ready to hear a million ways that social media has made conservative people more insane, but you hear any mention of how it has also deranged the left.

• Stolen Focus: Why You Can't Pay Attention, Johann Hari

  ⭐⭐⭐
  Another book by an extremely online journalist who documents his experience quitting the internet so he can focus again. He gets into the different things that our contributing to our inability to focus, with an emphasis on social media. He also veers off on some weird tangents, like when he rants about how ADHD isn’t real, and how pollution from cars is making you stupid. You can skip the chapters like this. I recommend this book solely because there’s a great section in the middle where he talks to Tristan Harris (an Google whistleblower) who explains in detail how social media sites are designed to be addictive. But to be honest, you can rewatch the Social Dilemma to get this information.
  

Ask most people these days and they would agree that we live in a world where things have gone wrong. If they’re my age, they might rattle off a sequence of historical events they have “lived through”. “Financial crisis, Trump, Brexit, Covid, russia-ukraine, israel-palestine.”

A lot of unfortunate people are really living through these issues, not experiencing them virtually. I am not calling these people doomers. My notion of the doomer is young, western, and educated. They often come from a middle class family. They are usually in white collar work or underemployed in the service industry. And they spend a lot of time ruminating on bad things currently happening (such as wars in distant lands) or large abstract trends in the world that don’t effect them on a day to day basis, like climate change. These people are engaging with these issues through social media. Or alternatively, by talking to people who consume a lot of social media (i.e almost everyone).

The Attention Economy

The attention economy has been called the economic incentive structure that is leading a “race to the bottom of the brain stem”1. It is a historical process that has coincided with modernity. Soon after the creation of the printing press, publishers realized they could sell attention grabbing pamphlets at a loss, because they could make more money from advertising than if they charged readers. This creates a perverse incentive, because advertisers then became the customers and consumers were the product. So instead of being incentivized to create the most entertaining or useful articles, they just started making whatever would get people’s attention.

Nowadays this mechanism has evolved from static media with attention grabbing content to the dynamic, personalized experience of social media. This has resulted in an increase in attention capture. Unlike newspapers or television, people always have their phones available to them. Legacy media may have demographics (the New-York Times reader is different to the Fox-News reader), but it is still broad brush. Your phone on the other hand, targets your specific combination of interests and prejudices with incredibly sophisticated AI algorithms running on the world’s most powerful supercomputers.

Most big platforms use the same algorithms with a little secret sauce thrown in. They are uniquely engaging due to advantages they get from their scale. First of all, they are able to curate large, high quality, proprietary datasets from user interactions across their services. Better datasets means better recommendation systems for engagement hacking. Secondly, they can afford the necessary computing infrastructure to train and run large models. This matters because AI is expensive to train and run. Thirdly, you need money to attract the best machine learning PhD graduates from Stanford, MIT, etc. And if you’re Meta, it costs you extra: reports says Meta must pay more for engineers because of the bad reputation they have.

The most aggressive and potent engagement hacking platform is tik-tok. Tiktok has a lot of advantages that the other big platforms have. In my opinion, their secret sauce to their addictive algorithm is probably the density of their training signal. In machine learning the training signal is similar to the general statistics concept of a “signal to noise” ratio. For example, if you have a computer vision algorithm that you are training to classify images of cat vs images of dog, if your dataset contains mostly pictures that have neither, it will have a “weak signal”. Indeed, one of the consensus heuristics amongst machine learning engineers is that it’s better to improve your dataset than improve your algorithm.

The New York Times described the TikTok algorithm based on leaked internal documents. In short, the algorithm optimises for engagement via two metrics: “retention — that is, whether a user comes back — and time spent. The app wants to keep you there as long as possible”. All social media apps do this. So why is tiktok so much better at it? It is because of its short video format, which has now been copied by instagram reels, youtube shorts etc. As an expert in this excellent podcast on the topic proposes, watching many short videos means the training signal is very dense. That is, in an hour of time spent on the platform, the personalization algorithm receives a tremendous amount of information about what will make you engage. Time spent on each video is a positive signal. Flicking off a video is a negative signal. Likes are a positive signal. Because of the shortness of the videos, it can receive much much more information per hour than another platform like YouTube.

Let’s say we measure information about user preferences in an oversimplified “thumbs up/ thumbs down” metric. For each video liked or disliked, we get one “thumb” of information. Compare two identical users. One watches youtube for an hour. In that time they might watch ten videos at six minutes each. That’s say, ten “thumbs up” of information. Now have that same user watching TikTok. They can watch a video in about 15 seconds. They watch it or swipe past (1 or 0). That’s 4 thumbs a minute. That’s 4 * 60 = 360 thumbs an hour. So the TikTok algorithm has a 36x stronger learning signal than youtube. As a result, the average user also spends more time on TikTok: 95 minutes per day as opposed to 40 minutes per day on YouTube.

Negativity Bias

So why does this process of attention engagement optimization lead to Doomerism? Because of negativity bias. To simplify it: people’s brains find bad news very compelling. As a rule, your brain is more focused on avoiding future pain than it is on gaining future pleasure. One psychology research lab put it like so “Social media algorithms are designed to ensure that you are delivered content similar to what you interact with on a daily basis. … Because we’re predisposed to focus more on negative content, we process it more carefully than positive stimuli, thus devoting more time to it.¹⁶ In doing this, (negative) news stories or sources get pushed into our feeds”2

Where Doomers go to hang out

A great example of the type of online community that best typifies this Doomer mindset is the forum “collapse” on Reddit, or r/collapse as the users call it. r/collapse describes itself as a forum where people can have “discussions regarding the potential collapse of global civilization”. In it, people post about income inequality, pandemics, nuclear war, and climate change. User’s posts typify almost every category of what psychologists call “unhelpful thinking styles”, which are thought patterns or “cognitive distortions” that are associated with mental health problems. Typical cognitive behavioral therapy for anxiety or depression aims to allow users to classify their thoughts into these categories and disengage from them instead of ruminating.

That so many terminally online people end up lurking on this subreddit is not an accident. It is the logical conclusion of the current economic incentive structures of the internet. For the terminally online doomer, every day is spent on the screen, focusing on terrible real or imagined events that they have no power to change. Instead of empowering people to go out and improve problems in the read world, the emotional affect of a Doomer is one of what psychologists call “learned helplessness”. Learned helplessness occurs when you are repeatedly traumatized by a bad event that you feel you have no control over. Then, when bad things happen to you in the future, you will not try to prevent them, even if you do actually have the ability to stop them.

One of the key attributes of a “doomer” mindset is the impulse to reduce complex issues to simple solutions or narratives. As a result, Doomers see people who don’t agree with them as being evil and stupid.

This plays out in the political sphere as voter polarization.

• Get fed negative information to make you feel afraid, angry and helpless about a contentious societal issue, let us call it issue “X”

• This is a result of the algorithm optimizing for your engagement with issue X

• You develop a oversimplified view of issue X due to the selection of information you see about it, whereas it is really a complex issue with many aspects to it, and it is reasonable for someone to hold the opposite view to you if they have different experiences or values

• Anyone who disagrees with your view of issue X is not just misinformed or stupid, they are evil and must be stopped at all costs

• Approaches to solving issue “X” flip flop between two extremes. Party A comes in and applies their ideological approach to issue “X”, which exacerbates it in one direction. Then party “B” comes in and applies the opposite approach, with a similar lack of success. Meanwhile the deep, complex causes of the issue remain ignored and it gets worse.

Sound familiar?

Reject Doomerism, log off and be useful

Rejecting doomerism is simple. Unplug your router for the weekend. Go out and leave your phone at home. Buy the newspaper if you want to keep up with current events. Better yet, read books about topics you want to be informed about. You will be more informed and less upset if you read a book about the history of Israel and Palestine, than if you scroll social media videos about it.

I will end with a quote from C.S Lewis, the British religious author who was called to give lectures to the public during WW2. For him, the reality of living through historical events was depressingly real. This quote is a good summary of how I think we should approach the problems of the world; with dogged persistence, optimism, and empathy.

”If we are all going to be destroyed by an atomic bomb, let that bomb when it comes find us doing sensible and human things.” - CS Lewis

As Kiwi readers already know, this year we had an election. A full agreement hasn’t been reached yet, but it’s looking like we will have a combination of National + ACT + New Zealand First. This article is about National’s incoming ban on phones in schools, and why, although it is good, it won’t solve the problem of adolescent screen use. For this, legislation restricting access to social media is needed.

Politicians promise things they don’t deliver on all the time. However with the phone ban policy, I suspect this will be delivered. This is because all it takes is passing the law. Its doesn’t require much additional institutional capacity or investment. They can tack enforcement of the ban onto other compliance audits they do with schools. Here is my summary of the policy, paraphrased from National’s website

• Student achievement in NZ has been in decline for 30 years

• Many schools here and overseas have experienced positive outcomes after banning the use of cell phones

• The details of implementing the ban will be left up to schools

• The ban will apply throughout the whole school day

• Parents can contact students via the school office

• Exceptions will be given to students who need their phones due to special circumstances (i.e health conditions)

It is worth noting that current policy allows schools to enact a ban like this themselves, however many do not as it is a hassle to enforce.

For example, in my hometown of Dunedin, Otago Boy’s High School enacted a ban which had positive results, with the rector(i.e principle) describing it as “one of the best things I’ve done”. According to the article, “The ban was brought in at the start of 2022 because (the rector) was noticing boys staring at screens all break instead of interacting and playing, staff were having to deal with too much cyber-bullying and there was no need for them as an educational tool… (the rector) said he had seen changes since the policy was implemented and it was now normal to see boys interacting and playing together during breaks. Staff were also spending a lot less time dealing with cyber-bullying.”

This ban is a great first step, but ultimately, it doesn’t go far enough if you want to solve the problem of adolescent screen usage.

Network effects create social lock in which can only be addressed with a blanket restriction

As someone who is looking to be a father in the future, I often think about what my screen time policy would be like with my own kids. Knowing about how these social media platforms are designed to maximize engagement makes me think the obvious approach would be a complete ban - no matter how much my kids complain.

However, I’m young enough to remember what it was like to be a kid. Being the only kid without a social media account or phone in this day and age would be a form of voluntary social ostracism. Maybe you can raise your kids to be resilient, and the benefits of being offline would outweigh this social penalty. But it would definitely be a struggle.

This is a network effect - because everyone is on social media, there is a large social utility for being on it. Or in this case, there is a social penalty for not being on it.

If you ban phones in school, this will make it easier for kids to focus, will reduce cyber bullying, and promote the development of better social skills. But this doesn’t stop kids from using social media outside of school hours, on the weekends and during school holidays, which are all big opportunities for kids to socialize. I remember fondly school holidays after year 10 and 11, spending my time skateboarding around, playing basketball, and meeting up with friends to hang out. We had Facebook at the time, but without smartphones it was mostly used to organize meet ups. It pains me to think kids are missing out on these critical experiences of socialization, which a phone ban will not address.

Merchants of doubt means the evidence is mixed - but the precautionary principle suggests a ban

Like most questions in social science, the academic picture of whether or not social media is bad for you is mixed. There are studies out there that point to little or no negative correlation between social media and mental health (which are cited extensively by social media giants). Social psychologist Jonathan Haidt maintains an excellent open source, collaborative review paper on the effect of social media on teen mental health. This is too large to fully dig into here but is worth checking out if you want a deeper analysis of the evidence (here is a great article he wrote talking about it.)

As suggestive as this graph is, we have to remember correlation is not causation. This is emblematic of a larger problem: talking about needing “evidence” of harm before we take action against social media plays right into their hands. It is very difficult to objectively measure something complex like well-being, and by the time we get good evidence of something we already know through common sense, much avoidable harm will have been done.

This is, incidentally, how tobacco and oil companies were able to delay their decline for so long. They employed paid scientific shills, i.e “merchants of doubt” to obscure the scientific picture. Then they said “you need overwhelming evidence of harm before you regulate us”.

Social media is a large divergence from how humans have always interacted with each-other. It is literally a historical anomaly. Never before have children (or adults) had their social interactions mediated by screens. Moreover, its not like social media is a neutral medium for communication (like text messages, which sort of emulate letters). Instead, your communication has piggybacking on top of it an attention hacking algorithm which is specifically designed to maximize your engagement with the service so it can build a model of your personality and target advertisements to you.

”I'm pleased with the strong engagement... Facebook just reached the milestone of 2 billion daily actives.. The progress we're making on our AI engine.. is a major driver of this.”
- Mark Zuckerberg, Meta CEO and Founder. (2022)

The real problem here is something called the attention economy. This is a business model in which user of a service (such as a social media platform or newspapers) are given access to it cheaply or for free. This isn’t out of generosity. Rather, in the course of using the service, you become the product. Your attention to the service is commodified and sold on to third parties who may want to know more about you, sell you something or modify your behavior. Often, it’s all of the above. Allowing children to be targeted by the attention economy is not good for their health.

The precautionary principle says when we approach radical departures from our tried and true modes of being, large evidence is needed of benefit and safety.

The burden of proof is on the social media platforms to show that their products are unambiguously safe before we allow children to use their services. I can’t stop adults using social media - it’s a free country and people do a lot of stupid things. But just like we protect children from other vices because they aren’t developed enough to handle them, we should protect them from social media. It is our moral duty to do so.

There is no reason why we should betray our children so that a handful of companies in a different country can get more ad impressions and post a higher quarterly profit.

Debunking a bad argument - “freedom” to consume and be addicted is not freedom.

An argument people use to reject an adolescent social media ban is that we it would be a restriction on children’s freedom to communicate. However if you think about it, this is the same line of reasoning that was used by capitalists before the ban on child labour was passed. At the time, industrialists (and weirdo economists) said a ban would be an outrageous encroachment on a child’s ability to voluntarily earn a living. However, with the benefit of historical hindsight, we can really see such arguments as self serving. The ban on child labour has protected children from the dissolving force of capital, which turns everything into a commodity. Let kids have a childhood. They can worry about working when they are adults.

In a household, adults restrict the freedoms of children for their own good in many different ways. Because of network effects, social media is a collective problem, and parents can’t solve it with their individual actions. It requires a literal “nanny state” approach - co-ordinated restriction (parenting) through the government, if you will.

This idea that freedom to consume is freedom is of course part of a wider ideological pattern. Because we live in an advanced capitalist society, the economy is premised on growth through consumption. The neoclassical economic framework that dominates our political economy is focused on the self interested, rational consumer maximizing their own utility by buying goods and services. This is in contrast to different modes of economic thinking that focus on other aspects of economic life, such as production, technology, and work. The way we frame the debate to focus on the freedom to consume is a political choice, influenced by an economic ideology we are all steeped in.

To put it in a broader historical concept, the ancients would laugh at us and say that being addicted is a bad form of freedom. Indeed, the religious concept of sin is that when we follow our baser desires and commit sin, it becomes more difficult to act in accordance with our higher values. Sin is not really about guilt ( a common misconception), but is about protecting yourself from behavioral patterns that will degrade you. It is a very modern point of view that the denial of consumption and pleasure is a form of oppression, and it is worth asking ourselves who benefits from this belief.

A sample social media regulation from Utah

According to the U.S News and World Report, which compiles statistics about various US states, Utah ranks first in the USA overall in quality of life. It is first in economic performance, with almost zero unemployment and high economic growth. It is first on financial stability, fourth in infrastructure, fifth in education and seventh in healthcare. Utah has some things figured out.

Utah is also forward thinking when it comes to the impact of technology on its youth. Recently they limited underage access to social media. The “Utah Social Media Regulation Act” (which kicks in on March, 2024) ensures that social media companies must do the following (taken verbatim from this official government information website)

• Verify the age of a Utah adult seeking to maintain or open a social media account

• Get the consent of a parent or guardian for Utah users under age 18

• Allow parents or guardians full access to their child’s account

• Create a default curfew setting that blocks overnight access to minor accounts (10:30 pm to 6:30 am) which parents can adjust

• Protect minor accounts from unapproved direct messaging

• Block minor accounts from search results

  
  Additionally, it puts the following restrictions on social media companies should parents consent to their children using their services:

• Companies cannot collect a minor's data

• Cannot target minor’s social media accounts for advertising

• Cannot target minor’s social media accounts with addictive designs or features

Overall, I think this legislation is a great example of a common sense regulation that the incoming government would be wise to pursue. Its emphasis on needing explicit permission from parents for their kids to use the service would break the chain of network effects. As soon as some kids don’t have social media, it is no longer a problem if your kid opts out. And when children do use the service, it protects them from its worst aspects like addictive features and night-time browsing. Overall I think this legislation is a good starting point for how National could approach a social media ban, which when combined with the school phone ban would undo a lot of the current harm that social media is doing to our children.

Introduction

Although recent hype is starting to cool, AI is still developing at a tremendous pace. In my last post, I talked about the transformer architecture. This architecture, which uses the attention mechanism, has been applied to achieve state of the art results in many different areas of machine learning. Many are finding success on a variety of problems by applying the transformer architecture to build “foundation models”. These are very large (10s of billions of parameters) transformer models trained on large amount of data. This training requires gigantic, expensive supercomputers. They gain a general ability in a given area (such as language, vision, audio processing etc). Through a process called fine-tuning, these generally capable models can then learn more specific business tasks downstream. For example, Meta recently released LLAMA-2, which is a language foundation model with general capabilities in language processing. With a few thousand high quality examples, this “base” model can be fine-tuned in a specific domain (such as legal analysis) which greatly improves its performance at that specific task at the cost of reducing its generality. This is an inherent trade off.

Because of the cost of pretraining a foundation model is prohibitive, only big tech companies can produce them. These models are usually locked behind some sort of pay per use API or chat interface (as in the case of GPT models) and heavily fine-tuned to make output more “useful” through reinforcement learning with human feedback (RLHF). Some compare the RLHF process to a lobotomy as we don’t know what it does to the capabilities of the underlying model. Although many people argue RLHF is necessary to make generative AI useful, recent research by Meta AI suggests that just about all of the capabilities are embedded in the pre-trained model and that only a few well crafted examples are required to fine-tune the model towards outputting in a useful format. There are also concerns that models are fine-tuned to reflect the worldview of their creators: rich, young, male, liberal or libertarian, atheistic, white/asian/indian San Fransisco based AI engineers. These people are definitely not representative of the general population of the world.

One of the most obvious lessons of 20th century history is that technological progress confers wealth to its corporate creators and military power to their countries. Joe Biden and Rishi Sunak recently met with industry leaders such as DeepMind, OpenAI, and Meta. China banned LLMs because they can’t RLHF them enough to stop mentioning Tienanmen Square. Because of what is at stake, there is fierce debate on how this technology should be developed and deployed, however broadly speaking, there are two main approaches; closed and open.

The current players and some recent advancements

Ironically, the biggest player on the closed side is “Open” AI, who is developing the GPT-4, chatGPT, etc. Open AI used to be a non-profit that was dedicated towards releasing all of its models open source to make sure that AI developed democratically. However since then they have taken a lot of funding from Microsoft and decided that their models are too powerful to release to the general public, although they are allowing Microsoft to use them in their unhinged Bing search model.

On the open science side, Meta (formerly Facebook) AI is the big player here. Meta AI has consistently released high-quality open source foundation models to researchers and the general public. Recently they produced SegmentAnything, an underrated and powerful computer vision foundation model, and the far more famous LLAMA which is a 65 billion parameter open source large language model similar to GPT3. Although they published a paper detailing LLAMA architecture (unlike OpenAI’s paper on GPT4 which was very light on info) they only allowed researchers to access its weights at first. Then one of the researchers promptly leaked it to 4chan. If you have the weights, you can now run it on a 2020 or newer MacBook locally using software called LLaMa/CPP (a rewrite of the inference engine in the faster language C++ instead of Python). In terms of quality its fine-tuned version (InstructLLama) it is slightly worse than the free version of chatGPT although through algorithmic advances such as QLORA have made it so that open source models are getting a better and cheaper to train. (Update: when I had already written this article, Meta released a new version of LLAMA, LLAMA2, which has an open source commercial license and is on par with Chat-GPT in terms of power.)

There is also Google, which doesn’t neatly fit into either bucket. They were the original creators of the paper that introduced the transformer model that powered this recent wave of AI, but they have also massively held back a lot of their capabilities, out of general concern for safety, but possibly also out of concern of undermining their own business model.

Some within Google are pessimistic about their ability to adapt. In the now infamous essay which was circulated throughout Google titled “We have no moat”, a software engineer bemoaned the rapid development of open source AI represented by the LLaMa series, LLaMa/CPP, QLORA, etc. Google is one of the most profitable companies in human history, but its dominance relies on people using its search engine to answer questions. For a lot of questions however, up to date information is not required, and it is possible (although I am skeptical) that in the future Large Language Models tendency to make shit up will be fixed, either through the models developing symbolic understanding, or through engineering methods, i.e integrating language models with extrernal databases so they “ground” their answer in facts. A good example is the recent model ChatLAW, which is able to produce legal advice with far less hallucination that its open source base model (LLAMA-1). To simplify how it works, the user’s query is passed into a language model, which extracts keywords. The keywords are then used to query a database, and the answers and references are passed to another LLM, which formulates an answer that summarises and cites the reference material. The user can then verify the citations, which creates trust.

So there are two possible avenues for solving the hallucination problem. I truthfully have no idea if either will succeed, but if they do, some people may stop searching for stuff online, as they can just ask their chatbots that they can run locally on their laptop. This could be a massive boon for end user privacy and productivity but could destroy Google’s business model. My personal view is that if it is possible to make it, a “Linux” of chat-GPT (i.e an open source, private.and hallucination free) chat-GPT clone would wrest a lot of control back from the hands of big tech and give it back to the average consumer.

Many tech commentators have said Google no longer has the institutional agility to effectively counterpunch. However, don’t rule out their recently absorbed subsidiary DeepMind, who has less of Google’s sclerotic bureaucracy and is still publishing high level research. In fact, they are currently preparing to counterattack with a Chat-GPT like model which also integrates some of the planning capabilities that made their AlphaZero/AlphaGo model so powerful. Time will tell if the new Google Brain / DeepMind partnership will retain the dynamism of DeepMind and what approach they will take. At this point I can see them going either open or closed depending on what Google wants.

Some arguments for and against openness

Releasers vs Controllers

In a recent interview with Lex Fridman, notorious hacker George Hotz outlined a certain view on openness in AI development, which I feel is a good summary of the popular case for openness you may see amongst what I call the “Hacker News” crowd. I will call those who are for open source development the releasers, and those who think we should develop in a closed and regulated manner the controllers.

In the releaser’s view, it unlikely that we stop AI development now. This is because we are in an arms race. He doesn’t go into too much detail on this, but I can expand. I think it is useful to think of this arms race as happening at two incentive scales; global and national. Of course, their are interactions between these scales, but it is a useful heuristic because it highlights the different incentives.

Firstly, within countries (America mostly), there is a commercial AI arms race.

Most powerful western AI companies (OpenAI, Google, etc) are in America which has a light touch for regulation as well as high regulatory capture and political corruption. We shouldn’t expect regulation to stop development as this would lead to curtailment of corporate profits

Because of the large financial incentives for developing powerful AI, an arms race condition has been triggered though venture capitalist hype, leading to investment. This investment will only be stopped by a massive reduction in hype due to product failures (think crypto’s arc from 2021-2022). However unlike crypto, there is a lot of empirical evidence that AI will meet at least some of its promise in delivering monetizable value and as such investment is unlikely to dry up and stop the arms race condition.

Zooming out from the financially driven, national arms race, there is a literal global arms race at the international scale driven by geopolitical conditions. This is the application of AI to warfare to gain military superiority. Some current, already existing examples are offensive and defensive autonomous drones, AI fighter pilots, missile defence systems, and automated influence operations.

Although there are entities such as the United Nations for controlling arms, they have had mixed success in the past, as viewers of the recent movie “Oppenheimer” can attest to. Personally amid the Ukraine war and recent tensions between the two global AI superpowers China & USA, I think this internation arms race condition will continue.

Therefore, we can assume more and more powerful AI development will continue. George also disagrees that we should worry about trying to control super intelligent AI or existential risk, which I agree with.

This isn’t because he believes these risks don’t exist, but because we can’t predict what the architectures of super intelligent AI will look like, assuming it is even possible. Therefore it doesn’t make sense to try to figure out how to control it now. It is akin to trying to figure out how to solve a math problem in the future, when you dont know what the problem will be asking, or if it will be algerba, calculus or number theory, etc. You can develop approaches that are generally helpful for controlling all possible upcoming AIs, but this may in turn cause an acceleration of AI development. A good example of this is recent research on understanding why AI models do what they do (in AI research jargon, this is called interpretability). The argument is that a greater understanding of neural network will give us a greater chance of controlling them in the future. However I think it’s just as likely a greater understanding of these models will lead to an increase in the speed of their development.

What releasers think you should be afraid of

George’s main concern is the massive power that capable AI will confer. As we have seen, technology confers power to those who can use it effectively. If AI development is highly controlled and stays behind an API like in the OpenAI case, that confers massive power to large companies like OpenAI and the governments which control them. How you feel about this depends on your view of these private and public institutions. For me personally, this seems like an undesirable outcome, as I have a low trust in governments and corporations.

So the main risk comes not from the typical example of terminator AI (or the nerd version, paperclip maximising AI). The real danger comes from the use of AI by bad humans, or even worse: humans with good intentions who feel a need to control others. In this scenario, open source AI allows people to use AI to defend themself from these bad AI and power is diffused instead of concentrated. For example, to defend against AI powered misinformation operations, you could have an AI browser plugin than can reliably identify and automatically filter propaganda while reading the internet.

The controller view

There are several strong arguments against the open development of AI. I think the best one hinges on the fact that it will lead to more powerful, ubiquitous AI faster, and that because of this, human power will be augmented in uncontrollable and unpredictable ways.

Open sourcing all models will massively speed up the development of AI. If you don’t think that us having super powerful AI soon is good, then this is a concern. For example, we don’t really have any good regulatory framework or social adaption in place to allow us to ameliorate whatever bad outcomes come with AI, and if we open source it and development is accelerated, we will have even less time to put things in place to allow us to adapt.
As a result of quickly improving AI, humans will have more intelligence. Because of economic and national security incentives, this will lead to more development of science and technology, which will further increase human capability, without also increasing out corresponding political, social, moral and spiritual wisdom. For example, what if AI allows anyone the capability to engineer a super virus 100x as bad as COVID? The invention of this technology need not coincide with an increase in our empathy and kindness.

Especially if AI is open source and cannot be monitored, criminals can use it to commit crimes more effectively, or repressive governments to commit acts of violence, etc. There may be an inherent asymmetry where an increase in AI capability makes it way easier for people to fuck things up, without making it way easier to defend people or heal them.

Thesis; Antithesis; Synthesis

As an AI engineer (albeit one working on applied projects, not capabilities), I think about the morality of open AI development a lot. This essay has been a snapshot in my thinking, based on an intuition I have developed through studying, working, and talking to other AI people. In the end, I think a balanced approach is needed. I came up with a simple maxim that sums up the situation we are in.

”Problems require co-ordination at the level at which they occur”

AI is a problem that will affect everyone. For it to develop in a way that benefits everyone maximally, I think we need global co-ordination to develop common sense, minimalist legislation. This would be an agreement between governments to not develop and deploy AI for military purposes, similar to the international ban on chemical and biological weapons.

Secondly, at the level of regulation of domestic AI, this is a poitical problem that is much more up to individual countries. Governments should decide the comfort levels that they have with the deployment of AI and will regulate AI deployment and access within their own borders, “great-firewall” style.

As to the titular question: is open source AI development good? I think it depends, mostly on the unfolding regulatory enviroment. If there is good regulation in the future that prevents misuse, open source AI development can continue apace, and people accross the world will safely apply it to positive applications such as medicine to benefit humanity. However if governments continue to rule out AI arms controls (as US, Russia, China etc have so far) and fail to regulate harmful commercial applications of AI such as engagement hacking I think it becomes unethical to develop open source AI capabilities, as you are contributing to the problem. I don’t buy the argument that open source AI will protect you from bad actors. It is a libertarian fantasy that you can just can outsmart bad institutions if you are good enough at tech, the reality is that they have so much more funding and momentum that the average person will never be able to protect themselves.

Housekeeping

Dear readers: I had a walk with some friends, and we had a discussion about this substack, which got me thinking. I have decided that subsequent posts will largely fit into two buckets:

1. Opinion pieces about the effect of technology on society These pieces will be on the topic that I think most about: the continued development of technology and the impact it is having on us, both as individuals and as a society. I will try to explain how we could approach technological development and deployment differently. I will also try to do original journalistic research into technological / social developments in my own country, New Zealand / Aotearoa

2. Technical explanations of topics in technology, mostly AI. This will expect a decent background in computer science, math and statistics and will not be aimed at a general audience. These articles will be based on reputable existing resources like textbooks or research papers. In them I will try to distill disparate resources about this topic into a single monolithic article that explains the topic as clearly as possible.

Accordingly, posts in the first bucket will be tagged (social) in the title, and posts in the second bucket will be tagged (technical). If I have something to say about the substack itself, i will tag it (meta) in the title, like in this post.

Here are my reasons for writing posts of both types, and why I think people should read them.

Social

• To bring attention to ways in which technology is impacting people and society, more specifically in my home country, New Zealand / Aotearoa

• To give specific recommendations about ways to avoid the negative effects of this at a personal level, based on my understanding of technology

• To give broader political / philosophical arguments about how to manage this at a societal level, based on my understanding of politics and technology

• I think people should read these posts because I think there is a lack of good writing about the impact of information technology on society from a view informed by both humanities and STEM, especially in the specific context of my home country, NZ.

Technical

• To solidify my understanding of a specific technical topic

• To give myself more motivation to do independent learning

• To practice my communication skills

• I think people should read these posts because I have a good combination of skill in technical matters and effective communication

• However I understand if people think there are better resources on these topics, in the end I am mostly posting these technical articles for self development purposes, and there are many great resources online explaining topics in AI or tech. So ignore these posts, or don’t, either is fine with me.

Examples of some ideas for articles I am developing that fall into these two categories:

Social

• Practical tips to resisting the attention economy

• The amish approach to technological adoption

• Would explainable AI be a cure all?

• A history of Luddite communities in New Zealand / Aotearoa

• The implications of the section 230 supreme court hearing on the NZ internet

Technical

• The basic theory of neural networks

• Computer Vision: Convolutional Neural Networks and YOLO

• Large language models explained (parts 2/3 and 3/3)

Thanks for reading.

Kia Kaha

• Ben

After a long break between posts due to grad school and general busyness, here’s my next post. This one is the first in a three part series on large language models. In this series I want to really drill down into the technology behind ChatGPT so people understand how it works.

In this post I explain the architecture behind models like Chat-GPT, the transformer. I explain the background to why transformers are successful and the key to the transformer architecture, the “self attention mechanism”.

The plan is for the second post to go over the other technical details of the GPT series - the architecture isn’t just a transformer! In particular, Chat-GPT and its predecessor Instruct-GPT have a reinforcement learning aspect to their architecture - they are basically a bigger version of GPT-3 that has been fine-tuned with human in the loop reinforcement learning to be good at chat dialogue.

Then for the third post, I will give my overall take on the LLM technology, its limitations, and what I think its future might be. This will be for a general audience.

A fair warning, the first two parts will technical and will requires some understanding of machine learning and maths. Non technical readers, feel free to skip the first two posts accordingly, although the second one will be less technical than this.

Background: Historical Context

Deep Sequence Modelling

Sequence data is a type of data that has a defined ordering, for example a book, time series data of stocks, and audio recordings are all sequences, of words, prices, and sound respectively. When people started to use machine learning to model sequence data, they used recurrent neural networks, which were created in 1986.1 Recurrent neural networks are feed-forward neural networks that have an internal "state" representation at each time step. Each state is dependant on the state at the previous time step. This gives RNNs the ability to model processes that change over time.

The problem with RNNs

However, RNNs have problems.

• Because they are inherently sequential, they cannot be parallelised like other neural networks, making them slower to train and run.

• RNN also lose information when modelling long sequences. This is because any given state only directly depends on the previous state. Thus information from states far away is lost

• RNN cannot learn bidirectional context, they can only predict a word based on the words that came before it, so they can't look ahead and see how a word fits in a sentence.

Lastly, they fail to train properly if you make them too long. This happens because of the vanishing gradients problem. This occurs when the gradients diverge as they are multiplied back over the long network during back-propagation. This is because the gradient of the state is dependent on the weight matrix and the derivative of the activation function recursively, i.e they are multiplied by them at each back-propagation step. So they are exponentially dependant on these values, and thus may decay or diverge exponentially, which breaks training for a long RNN. Let x_t denote the state at time step t, and x_k denote the state at step k, and W_rec the recurrent weight matrix. Then, we can see from the following equation the matrix product can explode or vanish when it is applied repeatedly during back-propagation if W_rec has a different magnitude than 1.2

LSTM to the rescue

This is where LSTM networks come in.

LSTM networks are a form of RNN which were created to solve the vanishing gradient problem RNNs suffer from. They are a complicated architecture, but essentially they solve the vanishing/exploding gradient problem by changing the gradient update step so that instead of being multiplied by a certain factor repeatedly, a factor is added. So the update operation is changed from multiplication to addition. This means the gradient can no longer vanish and is less likely to explode (although it still can in some situations). This allows memory to be retained for longer sequences. While RNN explode / vanish after ~10 timesteps, LSTMs can learn sequences of up to 1000 time steps.3

However, LSTMs still have a lot of the same problems RNNs do.

• Because of the inherently sequential nature they can’t be trained in parallel by a GPU.

• They are slow to run as well and can only process data sequentially.

• Furthermore LSTMs learn left-to-right context and right-to-left context separately, whereas in reality they are not independent.

So for a while, language models were sort of stuck. Sequential models like RNN and LSTM are inherently limited in terms of how efficient they can be: the deep learning revolution came about as a result of having methods that were able to harness the computation coming from the increasing development of GPUs (graphics processing units). These were developing at an ever increasing pace because people were using them to do wholesome activities like mining bitcoin and playing anime wife simulator at max settings. How did GPUs speed up neural networks? A brief foray into the basics of parallel computing is necessary.

If you have a task that can be broken up into many small subtasks that can be solved at the same time, and then put back together to give your overall solution, computer scientists say your problem is “parallelisable”. Certain problems, are considered “embarrassingly parallel”, because the speedup factor from this is very large, as the problem can be easily split into subtasks, and the subtasks don’t have to share information. GPUs are computer chips designed to do certain types of operations very quickly in parallel, which allows them to compute certain mathematical operations (like matrix multiplication) very efficiently. Computer graphics uses many of these operations to compute lighting and rendering etc, so as I said above, this is why GPUs were originally developed, people wanted games with better graphics. But the same kind of operations also occur in machine learning a lot. So the basic problem with LSTM and RNN is that we have these great big awesome powerful computer chips (GPUs) but because these architectures are not parallelisable (they are inherently sequential), we can’t get the full speedup that we can apply to other neural networks. That’s where the transformer comes in. The essential factor making the transformer highly successful is that it solves all of the problems with RNN, it is parallel (allows for bigger and faster models), it learns bidirectional context, and it doesn’t forget context as two words get further away in the sequence. It does this with two spicy mechanisms called self attention and positional encoding.

Transformers: Attention is all you need

Transformers are a neural network architecture introduced in 2017 by a team at Google Brain.4 Transformers are the “T” in Chat-GPT and other GPT models. They learn a mechanism called "attention" which allows them to learn to pay attention to different parts of the input sequence.

Critically, this new mechanism of attention allows them to learn relationships in a sequence in parallel. In RNN, positional information is encoded in a the sequential nature (i.e the state at N is related to the state at N+1 because it comes before it) whereas in the transformer, positional encoding is used to make sure tokens have information about where they are in the sequence without having to do sequential processing like in the RNN. Or as they say in the paper, “Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the token in the sequence”.

This makes them faster in two ways.

• While RNNs pay a O(N) penalty for processing sequentially, transformers parallelise naively in O(1)

• RNNs also pay a O(N) cost to correlate words N away in a sentence, while Transformers can learn correlations between far away data points with constant O(1) cost.

• They can also learn bidirectional context without learning "left" and "right" context seperately, meaning they can solve problems of the form "The glass of water is **** and it is sitting on the table.", i.e placing a word in a sentence.

In this way, Transformers solved the problems of efficiency and scalability that RNNs had, and were able to leverage increases in compute and data to reach state of the art on many different language tasks.

The key to understanding transformers: attention mechanism

Attention allows the network to focus on the most important parts of the input. Intuitively, attention works like search & retrieval.

• Identify which parts of the input to attend at

• Extract the features that correspond to these parts.

The attention mechanism does this with three variables: query, key, and value. Continuing with the search analogy, you can think of these variables in the following way:

• The queries are the term you are "searching" for

• The keys are the possible "matches" in the data set

You calculate the correlation / "compatibility" of the queries and keys by creating an attention mask:

Let me explain this equation a little:

• Softmax squashes the output so that it all sums to 1.0 (squishing it into probablity space, where likelihoods must add to 1.)

• Q and K are matrices made up of the individual query and key vectors.

• Thus the matrix multiplication QK^{T} is equivalent to taking the dot product of each query - key vector combination

• The dot product q dot k gives us a measure of the similarity of query vector q with key vector k

• d_k is the dimension of the key input (length of the keys). Scaling by this stops the dot product from exploding which would break softmax with vanishing gradients

In summary, what we are doing is calculating the similarity of each query with every key, and then doing some scaling to make it easy to work with. This gives us an attention mask.

You can conceptualise the "attention mask" just as you would conceptualise a mask on an image - high for pixels of interest (high similarity of query and key), low for irrelevant pixels (low similarity of query and key).

You then multiply the attention mask by the value matrix (the input values) to extract the part of the input you are paying "attention" to.

And that’s all attention is! It’s a bit of a complicated process, but once you get it, it seems simple. Maybe read this section a few times to let it sink in. I will also run through a numerical example. Again, if you are non technical, you can skip this part.

Self attention example

Say we have the following example for keys and values

And we have the following query matrix that consists of only a single query: (i.e a 1xn matrix)

We get the following result:

The query matches with the second key, so we get a 1.0 weighting on the second entry of the attention mask. Then, we multiply the attention mask by the value vector, extracting the 12.0 feature in the second row.

Let's try two query vectors this time. We still pass them as a matrix.

First we take the query vector

Equally matches the 3rd and 4th keys so we get an attention mask for the first part of

Likewise the second query vector equally matches the first and second keys so we get the following overall attention mask

This gives us the following attention

We can see the extracted features are the average of the 3rd and 4th key-values for the first query and the average of the 1st and 2nd key-values for the second query.

Attention in practice: “the ball is red”

OK, so that is how you calculate the attention of a given key, query, and value matrix. Now I will explain how this would work in an actual textual example.

Take the following string: “The ball is red”.

How do we convert this into the Q,K,V matrices? Well the Query, Key and Value matrices are generated by transforming an input feature matrix, X. The transformations the attention head performs to generate Q,K,V are made by three learned weight matrices, W_{Q}, W_{K}, W_{V}. So the model learns to calculate an accurate attention mechanism for a given feature input by training the weight matrices that generate our Q,K,V.

We could just as easily write the expression for attention as a function of the input feature matrix, and the three weight matrices.

Now let’s go back to the transformer. We’re not done yet, we need to convert the string into a series of vectors to make our Q,K,V. First we must give each word a word2vec embedding and positional information. The word2vec embedding is a way of representing words in a geometric space where similar words like "happy" and "glad" are closer together, and it is learned by running a neural network on a large textual dataset. We have to encode words as vectors because neural networks can't handle strings as input. As for the positional encoding, I described it above, it is essentially a way of encoding spatial information in the word vector. We then have the following sequence (Where E_{The} is the embedding of "The").

When this sequence X is passed into the attention head, it is multiplied with the learned weight matrices to generate the query, key and value matrices, which store the query key and value vectors for every word in the sequence in each row.

Now we pass the Q,K matrices into the attention mask function

This produces a normalised vector for each word in the sequence, where the ith entry in the jth vector is the "similarity" or attention mask coefficient telling us how much attention the jth word pays to the ith word. For our example above that would look like this

Then, we finally multiply this by the "value" vector to extract the final attention result for our feature input. A vector that represents the "attention" for each word in the sequence, where the ith entry is a product of the mask vector for the ith word with the values (which is the entry embedding transformed by the learned weight matrix W_{v}.)

Attention heads, multi-head attention

This process of calculating an attention mask overall represents a neural network called a “attention head” in the transformer model. Here the initial linear layer is our weight matrices W_V, W_K, W_Q. However in the paper many attention heads are trained and run in parallel, to learn to pay attention to several things at once.

More technically, instead of computing a single attention head with d dimensional k,q,v, they compute h smaller attention heads of dimension d/h. This way the model can learn to pay attention to a combination of things in the input. (In the paper, they say "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.")

Where h_n is the output of the nth attention head, defined as above by

Transformers for Translation - The "Attention is All You Need" Network Architecture

Now to put it all together, here is the original transformer architecture, taken from the paper "Attention is all you need" that introduced the transformer. Don’t worry if this seems super complicated, you don’t need to understand this diagram in full as the specifics of this architecture are related to the task the paper was addressing, language translation. I will go through and explain how this works using the concepts we have already covered, and this will give you an idea of how models like GPT use transformers to do natural language processing.

• Pass an English sentence to the input embedding.

• Input embedding converts each word into to word vector using a pre-trained embedding like word2vec so the network can process it.

• With a word embedding, words with a similar "meaning" are embedded close to each-other in a high dimensional vector space (usually around 64 dimensions)

• Each word embedding is then given positional encoding. This is a efficient way to store spatial context without having to do the recurrence operation like in the RNN and allows us to compute in parallel.

• After word embedding and positional encoding, word embeddings will be closer to each other in the embedding space based on the similarity of their meaning and how close they are to each-other in the sentence

• The word embeddings enter the encoder block, where they are input into the multi-head attention block

• Each attention head in the multi-head attention block uses learned weight matrices W_{Q},W_{K},W_{V} to transform the input word sequence embedding into a set of query, key and value vectors that form Q,K,V matrices

• The query and key matrices are used to calculate a "attention mask", a measure of similarity between each query and key combination, i.e a measure of similarity between every different word

• For example: the first entry in the attention mask of the first word embedding would be the similarity of the first word embedding with itself, the second entry would be the similarity of the first word with the second word, etc.

• This attention mask is scaled so that the self similarity of an entry with itself doesn't blow out the metric. It is then passed through the softmax function to normalize the distribution

• This attention mask is then used to extract an "attention value" for each word. For a given word this will be a weighted sum of each other word embedding, weighted by the attention mask between the given word and each other word.

• Here A_{i} is extracted attention value, M_{i,j} is mask between i and j, V_{j} is value corresponding to embedding vector of j (different than the embedding vector itself as it is multiplied by W_{V}).

• The result of these individual attention values is combined by the feed forward net to give a single attention vector for each word

• The attention vectors are passed into the decoder with the previously generated French word to predict the next French word

• The transformer translates a sentence from English to French by maximising the similarity of the attention structure of the inputted English sentence to the generated French sentence.

And that’s how transformers work!

Pat yourself on the back if you made it this far, because this is some heavy stuff. But now that we have this background out of the way, you will understand when I dive into explaining the GPT models and Chat-GPT in the next part of the series. Thanks for reading!

When people talk about AI nowadays, they are mostly referring to “neural networks”. At the moment there is a lot of discussion in and out of academia on whether or not these algorithms can be intelligent, including an ex Googler making the media rounds arguing that Google’s language models may be conscious.

In this article I give a quick overview of how modern AI works, and share my thoughts on why these algorithms aren’t intelligent now, and will probably not be in the future bar some unforeseen innovation. More practically, I also outline three immediate ethical problems that arise from this technology.

What are Neural Networks?

Neural networks are algorithms loosely inspired by the brain. A neural network is a program where a given input is passed through a set of layers of artificial neurons. Each neuron is connected to the neurons in the next layer through weights, which control how much the neurons effect each other. This is supposed to mirror how neurons work in the brain - certain neurons are more strongly wired to other neurons which is parametrised by the “weights”.

You can think of a neural network as a set of functions. Each layer is a function (f_i) that is parameterised by its weights to the next layer (W_i), and its input. So the first layer may be f_1(W_1,x) where x is the input (i.e an image we want to classify). If we have two layers, we can write the neural network as a nested function f_2(W_2, f_1(W_1, x)). The functions have a nonlinear component (called an “activation function”) so that the neural network can learn nonlinear patterns in the data.

It turns out that neural networks are universal function approximators: given an infinitely wide or deep neural network (a large enough number of layers), we can learn a function that maps any data input to whatever output we want. A common example of this would be a function that takes as input a picture of an animal (represented as an array of RBG pixels) and then maps this set of numbers to an output label: 0 (cat) or 1 (dog).

Neural networks learn these functions that recognise patterns in data by making guesses about the data, i.e you give it an input image and it guesses cat or dog. They improve their guesses by adjusting their weights every time they get a guess wrong. This changes the function to better map the input you give it to the correct output. At the start these guesses are random but, in the end, they learn the correct function to map the data input to the right output. An example could be an input picture of an animal, which would be mapped to an output label: 0 (cat) or 1 (dog).

This process of improving the weights to make better guesses is called back-propagation, because the error from the output layer is “propagated backwards” from the last layer to the first layer to determine how each weight is contributing to the error in the guess. This allows us to adjust each weight a tiny amount to reduce their contribution to the error and improve the guess overall. The math educator 3blue1brown has a good explainer of the theory behind back-propagation here. The math is relatively accessible for someone with a couple of college math classes, although the notation used is quite dense and takes a bit of effort to understand.

Are Neural Networks Intelligent?

As described above, in theory neural networks are universal machines capable of learning any function if you make them big enough. In practice however, they are quite limited. The universal function approximation theorem is more a mathematical proof based on a limiting case: in the real world we are not able to create an infinitely large neural network and give it an infinite amount of data. To train a neural network to do even a simple task requires a tremendous amount of data: they may have to see thousands of images before they can perform the cat/dog labelling I was describing earlier. In contrast to this, human beings have an internal model of the world and can learn new things after a few examples by comparing it with concepts they already understand.

Neural networks require a lot of data because of the type of reasoning they employ. In contrast to earlier logic based systems that performed deduction, neural networks reason via induction. In inductive reasoning, you infer a general rule from a pattern of observations. Whenever I see a swan, it is white. Therefore all swans are white. More formally we can say if we always observe A then B, we can say A implies B (or A=>B). When we show neural networks many example so that it can learn patterns in data, we are training it to make inductive inferences. This form of statistical inference neural networks use has been legitimately useful in science, engineering, etc. However it seems to fail when we try to use it to solve common sense problems. There is therefore a lot of scepticism among non “Big-Tech” affiliated academics that these systems can be intelligent.

For example: right now there is a lot of hype around “large language models” (LLM) like GPT3.  The architecture for these is complicated, but it relies on a type of neural network called a Transformer. Transformers are next word predictors: they learn efficiently which words relate to each other and use this statistical knowledge to predict the next word when you give them a sentence. If they have a big enough training set, LLMs can build up a decent statistical model of human language. They can then coherently mimic it. Ask them a couple of probing questions however and you will see they have no understanding of what they are saying: they have no world model or concept of causality. They fail when you ask them something outside of their training set. This is a more general problem that all neural networks have: their knowledge is brittle and narrow, and they cannot transfer it to new tasks.

Three Ethical Concerns

Firstly, these methods are data hungry which means that companies and governments are incentivised to collect massive amounts of personal data which is used to violate people’s privacy and manipulate them, like in the case of the Cambridge Analytica scandal. For a good popular treatment of this problem, I recommend the Netflix documentary “The Social Dilemma”.

Secondly, induction does not account for events outside the dataset, such as the proverbial black swan. This means these algorithms will have rare but catastrophic failures. Think of all the recent stories of self-driving cars crashing. When human drivers crash, they do so because of inattention, inebriation, or small mechanical errors. In contrast to this, self-driving cars often crash for no real reason, from a standstill, or while cruising down an empty street. Wikipedia has a great section of its page on Tesla criticism that has cartoon schematic drawings of the worst crashes showing the car’s trajectory. In these crashes you again and again see the consequences of the unreliability of neural networks: the Teslas accelerates into a tree of the side of the road, drives over pedestrians that have pulled over from car trouble, or just plain drives into a wall. Why the AI crashed in a given instance is inscrutable: maybe it saw a plastic bag flying by that it thought was a street sign.

Lastly the way these algorithms make decisions is opaque. They are a complete black box. You cannot ask the algorithm “why did you come up with this decision” because it is the result of the summation of millions of tiny calculations. This means that we cannot trust that the algorithms are making decisions for reasons we would be ok with. This is especially important when they are used to determine things like who should get a bank loan or who should be suspected of a crime, etc. This gets into the issue of algorithmic bias: harmful social biases that we do not want to reproduce may be present in the data we feed the AI which can make it discriminatory.

The truth of the matter is that AI as a field is probably a long way away from general intelligence. In the meantime, these algorithms have become more integrated with our lives, in ways most people aren't aware of. The aim of my newsletter is to write essays that help people understand what these algorithms are and what they are being used for. The application of AI won’t be democratic until the average person understands AI. Once this is the case, we can maximise the benefits and minimise the harm to the average person.

Subscribe now