Put one fact inside a stack of text and ask a model to find it. At 50,000 tokens, Gemini starts to miss it. The technical specification says it can take a million.
Mikhail Burtsev, an AI researcher who founded the DeepPavlov project at MIPT and works on transformers and memory in language models, runs this kind of test. A tokenised word is roughly one token, give or take, so a million tokens is a stack of books. His group's results say the model can hold the stack, but it cannot really read it.
That matters to anyone who has kept one chat going for an afternoon. At some point the answers get vaguer and the model contradicts itself. Burtsev's practical advice is plain: clear the context and open a new chat from time to time, because the model "starts to get tangled in its own thoughts" and cannot tell important from unimportant.
Why does it happen? Think of a transformer as having two kinds of memory. The first is long-term: everything it absorbed in training, from books to Reddit. It needs no attention to recall that. The second is short-term, or episodic: what it knows about your task and your conversation right now. This is the job of the attention mechanism, and it was built for short things. When attention was invented for machine translation, a sentence of fifteen words was typical, and thirty was already long.
Then the inputs grew. First 300 words, then 3,000, then 30,000. Modern models use many attention "heads" in every one of their layers, a hundred layers or so, each head looking at its own part of the text. But the weighing at the end still has to spread a fixed amount of attention over everything on the page. Stretch the page, and each piece gets a thinner slice.
Burtsev's own comparison is to us. Ask a person to find a word in a paragraph and it is easy. Ask them to find one in a whole book and they cannot hold the book in their head at once. Reading, they forget what came earlier. The model loses focus in the same way a person does.
So the team measured it. They hid facts in texts of growing length and asked the model about them. Past 100,000 tokens, or even 50,000, Gemini found the fact badly. Of its million-token window, it was effectively using 5 to 10% for a task like this.
Then the harder version. Give the model two facts and ask a question that needs both, so that neither alone is enough. The usable length shrinks again. With three facts to combine, the model needs no more than about two thousand words before it stumbles. A million on the box, a couple of thousand when the question requires thinking.
Burtsev's group published a version of this in June 2024. Their paper reports that popular language models use only 10–20% of their context effectively, and that performance falls sharply as the reasoning gets harder. A separate study led by Cheng-Ping Hsieh, posted in April 2024, found that of the models claiming context of 32,000 tokens or more, only about half held up at 32,000.
Is the cure a bigger window? Burtsev goes the other way. His group's recurrent memory transformer keeps the window small, about a thousand tokens, and cuts a long text into a thousand such pieces. After each piece, the model writes a short note into special memory tokens, which carry over to the next piece. Attention never has to stretch, so it never blurs, and the computing cost stops growing with the square of the length.
With this, they answered a question over 50 million tokens. That is five times the Encyclopaedia Britannica, Burtsev says, and fifty times Gemini's window. The price is that the model gives up some generality, since it cannot squeeze everything into the notes. Nobody has run a neural network over more than that, he says, in the two years since.
The blame does not lie with the engineers who built the million-token models. It lies with a part of the machine whose job is to weigh everything and which, given too much, weighs nothing well.
If your sequence is very long, all your attention smears out, and you lose focus. The model loses focus exactly the way a person does.
Если у вас очень длинная последовательность, то всё ваше внимание расплывается, вы теряете фокус внимания. Модель точно так же теряет фокус внимания, как и человек.
— Mikhail Burtsev, Think Like a Scientist, 74:53
Nobody has even tried a window much longer than a million, because few real tasks seem to need it.
