In 1948, Claude Shannon generated English with nothing but a book and a rule. Open the book at random, pick a word. Open it again, find that same word, and write down the word that follows it. Repeat.
What comes out sounds like English for a few words at a time and then wanders off. One of his samples reads: "THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED."
That is a language model. It predicts the next word from the last one, using counts from real text. Look at more previous words and the output gets better. This family of models, called n-gram models, ran most of the world's speech recognition and machine translation for about twenty years. Then neural networks replaced them almost completely.
This post is about why, and about what counting still does better.
The wall
In 1956 and 1957, Noam Chomsky argued that no model of this kind can fully capture English. An n-gram model only ever sees a fixed window of recent words. Mathematically, that makes it a finite-state machine: it can be in only finitely many situations, and it forgets everything outside the window.
Language does not respect windows. "If" waits for its "then." "Either" waits for its "or." A name mentioned in the first sentence of a story can come back forty sentences later. These links can stretch as far as the writer likes, and they can nest inside each other. A fixed window cannot follow them in general, however large you make it.
The fix that almost worked
The bigger problem in practice was more boring: most word sequences never appear in any corpus. Count five-word phrases in a billion words of text and nearly every new sentence you write still contains a five-word phrase nobody counted. A naive counting model gives that sentence a probability of zero.
Decades of work went into "smoothing": sharing probability from seen sequences with unseen ones. The best-known result is Kneser-Ney smoothing (1995). Its key idea is simple. The word "Francisco" is common, but it almost always follows "San." So when you have to guess a word in a new context, "Francisco" should be a rare guess. Kneser-Ney asks not "how often does this word appear?" but "in how many different places does it appear?"
A careful study by Chen and Goodman in the late 1990s found a modified version of Kneser-Ney to be the strongest smoothing method they tested. Counting also scaled. In 2007, a Google team trained a counting model on two trillion words and found that at that scale even a crude method called "stupid backoff" worked very well.
Why neural networks won
Counting treats every word as an unrelated symbol. To a counting model, "cat" and "dog" are as different as "cat" and "democracy." Everything it learns about one says nothing about the other.
In 2003, Bengio and colleagues gave each word a learned vector, so that similar words land near each other. Evidence about "the dog slept" now helps with "the cat slept." That sharing is what counting lacked.
Attention, in 2017, then removed the window. A transformer can look back at any earlier word directly. Similarity plus unlimited reach is, roughly, the recipe that took over the field.
What counting still does better
Counting was dropped. It was never disproved. Some of its properties are still hard to get any other way.
Adding knowledge is cheap. New text means new counts. Nothing is retrained, and nothing already learned gets overwritten.
Every prediction is traceable. You can point to the exact sentences a prediction came from. That matters when you need to know why a system said something.
Forgetting is exact. Remove a document's counts and it is gone, not just made less likely.
It runs on almost nothing. Looking up counts needs no special hardware.
The field has noticed. In 2024, the infini-gram project indexed over a trillion tokens with suffix arrays. That let an n-gram model use a window as long as any matching text, and its predictions still improved strong neural models when the two were combined.
The open questions
The history leaves some questions that seem more interesting than "which model wins."
Is the gap between counting and neural models mostly about similarity, or mostly about memory? These are different problems. Similarity can come from methods that are not neural at all: latent semantic analysis (1990) found word similarity using only counting and a matrix factorization.
Can a system keep track of things far back in a text without an unbounded internal state? Chomsky's argument says a fixed window alone cannot. It does not say which kinds of extra state are enough.
How much of what we call fluency is really local? If most of what makes a sentence read well lives within a few words, then the hard part of language is somewhere else, and it is worth being precise about where.
The history of AI is full of ideas that were set aside when something better arrived, not when they were shown to be wrong. Counting is one of them. The reasons it lost are well understood. Whether those reasons are permanent is still an open question.
References
Shannon, C. E. (1948). A Mathematical Theory of Communication.
Chomsky, N. (1956). Three Models for the Description of Language. (1957) Syntactic Structures.
Deerwester, S. et al. (1990). Indexing by Latent Semantic Analysis.
Kneser, R. and Ney, H. (1995). Improved Backing-off for M-gram Language Modeling.
Chen, S. F. and Goodman, J. (1999). An Empirical Study of Smoothing Techniques for Language Modeling.
Bengio, Y. et al. (2003). A Neural Probabilistic Language Model.
Brants, T. et al. (2007). Large Language Models in Machine Translation.
Vaswani, A. et al. (2017). Attention Is All You Need.
Liu, J. et al. (2024). Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens.