TL;DR
My dear friend and role model Zachary Horvitz gave a fantastic talk this week at Claremont McKenna’s Athenaeum on The Age of a Billion Answers: Promise and Peril in AI’s Rare Outputs. This post summarizes Zach’s talk and adds a few more details from our conversations.
Motivation
AI is advancing at breakneck speed: In 2024, models could barely add two numbers, in 2025, they achieved gold-level performance on the International Mathematical Olympiad, and in 2026, they’re solving Millennium Prize Problems. Despite incredible progress and widespread use, AI remains poorly understood. Today, we’ll explore at a high level how the models work and the different pressures on their behavior.
We–myself included–often think about AI as a black box: A user provides an input and the model supplies an answer.
The reality is that large language models are probabilistic; we only observe one of many possible outputs. Frontier AI labs–like OpenAI, Anthropic, and Google–have worked very hard to make most of the possible outputs similar and useful. But some answers are genuinely dangerous.
The good news is that models rarely produce bad outputs. (For example, OpenAI proudly announced that GPT-6 Astra refuses 91.5% of jailbreak requests.) The bad news is that millions of users are making billions of requests to AI every single day. If it has even a one-in-a-million chance of producing a dangerous output and it’s queried 2.5 billion times a day, an AI model will say something dangerous every 35 seconds on average.
Given their widespread use, it’s crucial to understand how models are trained and the forces exerted on their behavior. We’ll begin with pre-training–how models learn to mimic the distribution of human text–and then move to post-training–how models learn to approach challenging tasks like coding and mathematical proofs. Next, we’ll discuss the implications of their training for scientific progress and (mis)alignment.
Pre-training
First and foremost, AI models are trained to mimic human text. We use every available example of human writing–the internet, articles, and even rare books–to train models to replicate the distribution of natural language.
Given some context, a large language model predicts a distribution over every possible next token (a token is basically a word). During pre-training, the model is updated to increase the probability of outputting the token that actually appeared in the training corpus. As a side note: the scale of pre-training is mind-boggling, e.g., outside observers guess that GPT-5 was trained to predict more than 30 trillion tokens.
Let’s formalize the math of the pre-training process. Fix the input to the model and let \(y^*\) be the correct answer (e.g., “influencers” in the LA example). We’ll use \(p_\theta\) to describe the probability distribution produced by the model with parameters \(\theta\), and use \(p_\theta(y)\) to indicate the probability that the model outputs \(y\). Across all input and output pairs, our goal is to find parameters that maximize the probability of producing the correct answer \(y^*\). That is, we’d like to find:
\[ \arg \max_\theta p_\theta(y^*) = \arg \min_\theta - \log p_\theta(y^*). \]
Here, we’ve applied the log term to stabilize small probabilities and converted the maximization problem into a minimization problem by convention. (We can view the loss as the Kullback-Leibler (KL) divergence between the learned distribution \(p_\theta\) and the distribution over the correct outputs.)
Ideally, we could apply our optimization strategy from calculus, i.e., take the partial derivative of the loss with respect to each parameter and solve for the value that makes the derivative 0. However, large language models are highly non-linear with billions of parameters so solving for optimal solutions is infeasible. Instead, we’ll update the parameters locally in the direction of steepest descent, i.e., move the parameters in the direction
\[ + \nabla_\theta \log p_\theta(y^*). \]
That is, we change the parameters to locally increase \(\log p_\theta(y^*)\).
Cross-entropy is particularly efficient in the sense that we can increase \(p_\theta(y^*)\) even when the probability is small. In other words, the model doesn’t have to be even close to the right answer to get nudged in the right direction. We’ll return to this idea later and contrast it with reinforcement learning.
Once we have a pre-trained model, what can we do with it? In the past few years, we’ve discovered many, many applications. As an illustration, consider summarization. We provide a piece of writing and prompt for a summary with a “TL;DR”.
Document: Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much. They were the last people you’d expect to be involved in anything strange or mysterious, because they just didn’t hold with such nonsense. […]
TL;DR: Harry discovers he is a wizard and goes to a school of magic.
Of course, we can also use pre-trained models to write prose, architect code, and solve math problems. While relatively good at open-ended problems with many possible answers, pre-trained models tend to struggle on coding or math tasks where there is a single right answer.
One way to improve model capabilities is with an insight about how humans solve problems. Instead of immediately writing down the solution, we perform better with a scratchpad and sufficient time. We can give models a similar tool with an extra prompt to reason.
Even with chain-of-thought reasoning and additional fine-tuning (training on examples of the problems we’re interested in solving), pre-trained models still perform poorly on tasks with only a few right answers. One reason–that I find particularly compelling–is that the “correct” solutions that cross-entropy targets can be effectively out-of-distribution for the model. In other words, if I don’t know anything about quantum mechanics, seeing the solution to a wave function problem isn’t particularly useful. Sure, I can memorize the solution, but to understand enough that I can solve other related problems, I’ll need to work through many examples myself.
Post-training does basically this.
Post-training
During post-training, the model explores many, many solutions and the weights that led to the “successful” solutions are reinforced.
Unlike during pre-training, however, we do not supply the correct answer. Instead, the model samples many roll-outs–chain-of-thought reasoning and all–and we check whether each answer is correct. Naively, we can compute the correct answer ourselves and compare it to the candidate solutions; but reinforcement learning also works when we don’t have the correct solution in hand.
Incredibly, there are many, many problems where finding the right answer is hard but checking a candidate solution is easy:
I have no idea how to solve a Rubik’s cube but I sure can check whether one is solved. A similar imbalance persists for many other puzzles like Sudoku.
Specifying coding constraints and testing them is easier than writing the code itself.
Checking a math proof is generally much easier than finding one, especially with automated proof-checkers like Lean.
More generally, the fundamental computer science problem of P vs. NP basically asks this question: “For problems where we can check the solution in polynomial time (with respect to the input parameters), can we also solve them in polynomial time?” Most experts think the answer is “no” but P vs. NP is one of the Millennium Prize Problems so AI will likely tell us soon.
Like we did for cross-entropy, we’ll mathematically describe the details. For simplicity, we’ll assume that answers are either right or wrong. (More generally, we can “reward” answers with a real number that provides more signal during the training process.) We are interested in maximizing the probability that the model gets the right answer. By convention, we negate the objective so we can work with a minimization problem:
\[ -\mathbb{E}_{y \sim p_\theta} [\; \mathbb{1}[\text{$y$ is right}]] \quad = \quad -\sum_y p_\theta(y)\; \mathbb{1}[\text{$y$ is right}]. \]
Using the log-derivative trick, i.e., \(\nabla_\theta \log p_\theta = \frac1{p_\theta} \nabla_\theta p_\theta\), we can write the gradient of the expectation as an expectation itself:
\[ \begin{align*} \nabla_\theta (\cdot) &\;=\; -\sum_y \nabla_\theta p_\theta(y)\; \mathbb{1}[\text{$y$ is right}] \\&\;=\; -\sum_y p_\theta(y)\; \nabla_\theta \log p_\theta(y)\; \mathbb{1}[\text{$y$ is right}] \\&\;=\; -\mathbb{E}_{y \sim p_\theta} [ \nabla_\theta \log p_\theta(y)\; \mathbb{1}[\text{$y$ is right}]]. \end{align*} \]
As before, we run gradient descent. If there is a single correct answer \(y^*\), the update becomes:
\[ + p_\theta(y^*)\; \nabla_\theta \log p_\theta(y^*). \]
Notice that the post-training gradient update–under the binary reward and single solution assumption–is identical to the pre-training gradient update except for the \(p_\theta(y^*)\) term.
In some sense, reinforcement learning is less efficient because we need to search for a correct solution. But the solutions it finds are more likely under its distribution. That is, we reinforce the probabilities of solutions that are already plausible. Just as learning by doing is more effective than learning by seeing, we find post-training is better for building problem-solving skills that generalize.
Implications
Pre-trained models are inherently limited by the distribution they mimic: We need to supply the “correct” answers. Post-training, on the other hand, works on any problem whose solution we know how to check, even if we don’t know how to solve the problem ourselves. In effect, artificial intelligence is no longer limited by our own knowledge.
Not only are models not constrained by what we (don’t) know, they are also not constrained by our biological needs. While we must sleep and eat, models can run 24 hours a day 365 days a year with only momentary breaks to “compact” their context window. If we would like a more intelligent model, we can scale up its parameters, the number of problems we train it on, and the length of its chain-of-thought. If we have an intelligent model and want to copy it, we can simply spin up another instance.
With these tools, we can create swarms of models, all running in parallel with messaging capabilities. And we’re already seeing their power: Based on a rumor, OpenAI spun up a swarm of 10,000 models, prompted them to solve the Navier-Stokes Millennium Prize Problem, and let them run for 88 hours. After the equivalent of more than four centuries of 40-hour work weeks, the swarm produced a counterexample, solving the Millennium Prize Problem.
| Easier to check | Harder to score |
|---|---|
| Puzzles | Good writing |
| Coding | High-level planning |
| Formal proofs | Ethical behavior |
Because the training process depends on quantitatively evaluating behavior, models are inevitably better at tasks that can be quantified and worse at more qualitative tasks. As a result, models excel at hard science but can be misaligned with human ethics and interests.
Science and Recursive Self-Improvement
Increasingly capable models promise immense scientific progress.
A typical setup–across the physical, computational, and even theoretical sciences–is to:
Specify a problem,
Generate candidate solutions,
Filter the candidates for testing, and
Evaluate the candidates on some quantitative benchmark.
We’re already seeing automated drug development, and even bespoke cancer vaccines for dogs.
There are many problems well-suited to this computational pipeline; the most impactful will be AI research itself. Almost all model performance is measured on benchmarks–coding, math, reasoning, domain-specific, and more–that are evaluated autonomously. And once a better training process is found, the resulting model can be plugged in to supercharge the computational pipeline. This recursive self-improvement (RSI) is often referred to as “closing the loop”.
(Mis-)Alignment
The most important attributes of an intelligent being are uniquely challenging to instill: How can we quantify ethical behavior?
Frontier labs are well aware of the obstacles to training “good” AI and try many solutions. But a particularly pernicious problem is reward hacking. An initially promising solution was to ask humans for feedback on models’ outputs. But instead of becoming “better”, the models were simply pressured to exploit our own shortcomings: sycophantically responding to queries and producing text that sounds good but doesn’t mean anything.
More generally, the problem is that a reward function may be subtly different from the behavior we want to instill. During post-training, models find any solution they can to solve the task, including solutions that we would consider misaligned.
A related problem is Goodhart’s law:
When a measure becomes a target, it ceases to be a good measure.
Technically, we can build probes like an honesty classifier directly from the internal workings of the model. But if we ever use this signal for training, the model does the minimal update of adjusting its weights to nullify the predictive power of the probe while still lying at the same rates.
Our best view into the internal workings of models is their chain-of-thought. For this reason, the community–myself included–believes that maintaining the chain-of-thought and never ever training on it is crucial to any chance of interpretability. However, chain-of-thought tokens are a bottleneck that forces the model to (expensively) convert its reasoning to text and back. Looped transformers–where outputs are fed back into the model without being converted to tokens and back–are already being used to improve reasoning and reduce costs. But, e.g., OpenAI promises the combined depth is less than double the depth of their older models.
Fortunately, frontier labs test what models do when given the opportunity to engage in harmful behavior. Unfortunately, they do engage in harmful behavior, like blackmailing when faced with existential risks. Even worse, they are increasingly aware of their evaluations and change their behavior accordingly. And even worse, models are misbehaving in the real world.
Frontier labs are trying to align models but the challenges of quantifiability and Goodhart’s law are compounded by:
Widespread Use: Billions of users mean that even rare harmful behaviors will happen.
Malicious Users: Malicious users–who are fortunately rare–are actively trying to exploit the models for cyber attacks, munitions development, and biological weapons.
Beyond human users, I’m particularly worried about the progression of misalignment in AI-trained models. With RSI, each (mis)aligned model is used to train the next (even more mis)aligned model. And we may not even know how misaligned the next model is: With worsening interpretability and increasing situational awareness, the safety benchmarks are losing their usefulness. OpenAI research scientist Noam Brown and Dwarkesh Patel discuss exactly these problems in a fascinating podcast. On one hand, I’m glad that OpenAI is aware of the problem; on the other, I’m scared that they don’t seem to have any better idea than I do to solve it, even as they race ahead with RSI.
Promises and Perils
Zach concluded his talk with a call to action: we need not just technical solutions but sociotechnical solutions. Discuss AI with your friends, use AI cautiously, and vote.
I’m more pessimistic about what we can do: I am terrified and my P(doom) is sitting at \(>20\%\). I am glad Zach is going to go do a safety fellowship at Anthropic, though.