Hide one fact inside a long text, then ask a language model to find it. Mikhail Burtsev, an AI researcher who founded the DeepPavlov project at MIPT, runs this test again and again, making the text a little longer each time. Gemini can take a million tokens on paper. Past 50,000 or 100,000 it already fumbles the fact. For this kind of task it uses only 5 to 10 percent of the length it was built for.
Make the question harder and it gets worse. Ask something that needs two buried facts combined, and the usable length shrinks again. With three facts, the model copes with no more than about two thousand words.
You have met this in your own chats. Sooner or later you open a new window, because the model has started to tangle itself in its own thoughts and can no longer tell the important from the unimportant. There is a second bill you never see. Every word in the text is compared with every other word, so the work grows as the square of the length. Longer context means more electricity, more memory, and slower answers.
Both problems come from the same place. Attention is the transformer's short-term memory, the part that holds what is happening in this conversation right now. Stretch it over a million tokens and it spreads thin. It is like being told to find one word in a whole book: you cannot keep the entire book in your head, and reading, you forget what came before.
The fix came sideways. Burtsev's group was not trying to stretch anything. They were thinking about thinking.
A transformer does its thinking inside a vector, a long list of numbers that gets updated as it computes. That is its working memory. He wanted the model to hold several such thinking spaces side by side, to try alternative hypotheses, and to glance from one into another. So he added extra tokens to the input, tokens with no link to the meaning of the text, a blank scratchpad the model could use however it liked. Machine translation improved a little.
Then came the turn. A recurrent network looks at its own state from the previous step. Give the transformer a scratchpad and it can do the same, one whole chunk at a time.
We have a short context, say 500 words, and we added memory to it. When the transformer has thought this context over, it creates, in parallel, a representation in memory that corresponds to it.
У нас есть контекст коротенький, например, 500 слов. И мы к нему добавили память. Когда трансформер обдумал этот контекст, он параллельно создал некоторое представление в памяти, которое ему соответствует.
— Mikhail Burtsev, Think Like a Scientist, 82:28
That representation goes in as input for the next chunk. The model reads chunk two with its own earlier thoughts in front of it, updates the memory and passes it on.
So a million tokens stop being one giant window. They become a thousand pieces of a thousand tokens. The model never holds more than one piece, so the memory it needs stays the same however long the text gets. Attention always works inside a range of zero to a thousand and never stretches. Wait long enough and the text can be as long as you like.
They tried it. The model answered questions over 50 million tokens, which is five times the Encyclopaedia Britannica and fifty times Gemini's window. A paper from Burtsev's group, written by Ivan Rodkin in July 2024, reports 79.9 percent accuracy on single-fact questions at that length, on a benchmark called BABILong. At a million tokens, Burtsev says, their model scores 95 percent against Gemini's 20.
He is quick to take the shine off his own number. Their model is a specialist and Gemini is a generalist, and a tuned Gemini would do well too. The memory also costs something: the model cannot squeeze everything it reads into it, so it loses some of its versatility. It works for specific tasks.
When the episode was recorded, no neural network had processed more than 50 million tokens, two years after the record. The big labs could stretch a transformer that far with a cluster and a lot of engineering. Nobody tried. Burtsev's own guess is that nobody needs it, and that even a million tokens may be more than real tasks ask for.
What he had, in his words, was a hammer with no nails. So they went looking for nails, and found a very long text already sitting in every cell of your body. The human genome is about 3 billion letters. Cut into DNA tokens of roughly ten letters each, that comes to around 300 million tokens, against Gemini's one million.
