Introduction
We are trying to train a language model without gradient descent, backpropagation or Adam. Instead of taking thousands of small noisy steps, our solver takes a few exact Newton steps computed from closed-form statistics of the data. On a small training set (30,000 sentences) it beats Adam. As the data grows, it falls behind: about 0.15 nats per token behind at 100,000 and 300,000 sentences, and about 0.19 at the full 2.1 million. This post is about why, because the reason turned out to be more interesting than we expected.
The first surprise: a speed limit
The first surprise was a speed limit. An optimiser that takes steps of a fixed size cannot rest in a part of the loss landscape that is too sharp. For Adam, that limit can be written down: the sharpness, measured in Adam's own units, has to stay below 38 divided by the step size. We measured it on both models. Adam's solution sits at 14.6, inside the limit. Ours sits at 285.6, seven and a half times past it. Adam could not stay in our solution even if it landed there. Exact Newton steps have no such limit, so our solver happily takes the sharpest, deepest minimum it can find. On training text that is a great place to be, and on text it has not seen it is a worse one.
Attempting to respect the limit
That suggested an obvious fix: make our solver respect the same limit. We tried six ways. We penalised the sharpest direction, re-set the penalty strength at every step, applied the rule early and released it, applied it late, and applied it at three different data sizes. The sharpness did go down, by as much as 60 percent. The held-out score moved by at most 0.005 nats. Pushing down the sharpest direction just hands the sharpness to the next one, and a penalty halves the progress of every step. Our reading is that this limit comes from taking many small explicit steps, which a method built on exact steps does not do, so a penalty copies only part of it. We state this carefully: our measurement shows the limit is violated, not that violating it is the only reason for the gap.
The second finding: linear solver inefficiency
The second finding was a surprise in the other direction, and it came from a place we had stopped looking: the linear solver inside each Newton step. Each step asks us to solve a large linear system. We solve it iteratively, and a good preconditioner can make that much faster. We had been using a simple diagonal one. We replaced it, for the role maps that carry most of the curvature, with a structured one built from exact statistics of the data (a Kronecker-factored form), solving the same system as before. On a 100,000-sentence test, it reached in 3 iterations the quality the old version reached in 24, and its 24-iteration answer was far better. Applied as a real step, the improvement in held-out score was 0.056 nats against 0.0095 for the old solver. In other words, much of what we had blamed on the shape of the loss landscape was our solver not solving its own problem. We have measured this at 100,000 sentences. We are now running the same test at the full 2.1 million, and we will report it whichever way it goes.
Where this leaves the work
Where this leaves the work. Our best system today combines count tables with the exactly fitted model and scores 4.146 nats on our key test cells, against 4.038 for a same-data transformer, a gap of about 0.11. We do not beat the transformer, and this post is not a claim that we do. What we can say is that two things we assumed about the gap were wrong, one in each direction, and both were found by measuring rather than guessing.