A million tokens is about as much as the best chatbots can hold in mind at once. One human genome, cut into words of ten letters, comes to 300 million.
Mikhail Burtsev, an AI researcher who founded the DeepPavlov project at MIPT, had just built a transformer that carries a memory from one chunk of text to the next. It answered questions over 50 million tokens, which is five times the Encyclopaedia Britannica. Then came the awkward part. He had a hammer, and language processing had no nails left for it. So you have to "run around with this hammer" until you find something that needs hitting.
The nail was already in every cell of your body. DNA is a string of four letters, and that string decides a large share of the risk for the diseases that kill millions of people a year. Burtsev put the inherited share of those diseases at 30 to 80 percent. Today's methods explain about half of it.
What would a model that reads the whole string give you? Two things he names. The first is a dose. People break down the same drug at very different speeds: one takes three hours, another ten. A drug built to release slowly over ten hours hits the fast person with a much bigger dose than intended, and the side effects follow. That speed is often set by genes, so a model that reads a person's genome could predict the dose.
The second is gene therapy. A virus carries a useful gene into the cell, and you want it switched on in the liver and silent in the brain. The switch is a stretch of DNA placed in front of the gene. The virus has little room to carry it, so the stretch has to be short and well chosen. A model that has learned the regulatory logic of the genome can search for those sequences.
First, though, the text has to be cut into pieces. Reading DNA one letter at a time wastes the context window: you could train a chatbot on single letters too, and nobody does, because words fit more text into the same space. So language models use a method that builds a vocabulary from the most frequent strings of letters. When it cuts a text, it tries the longest entries first and fills the gaps with shorter ones. The vocabulary size is a trade. A bigger one squeezes the text shorter, but at every step the model must pick from more candidates.
Burtsev's group took the method and pointed it at DNA.
We simply took this approach and applied it to DNA. We wanted our model to eat more DNA. And we set the dictionary size at 30 thousand tokens.
Мы просто взяли этот подход, применили к ДНК. Мы хотели сделать, чтобы наша модель больше ДНК съедала. И мы задали там 30 тысяч токенов размер словаря.
— Mikhail Burtsev, Think Like a Scientist, 93:02
The 30,000 most frequent motifs became the vocabulary, and the average token came out at 9.5 letters. In the group's paper in Nucleic Acids Research in January 2025, led by Veniamin Fishman, the dictionary was 32,000 entries and the median token was 9 letters, enough for input of up to 36,000 base pairs.
Now the twist. Do those tokens mean anything in biology? Burtsev's answer is that nobody knows yet. A motif that keeps repeating may have been kept by natural selection, so it should matter. But run the same method on the genomes of different species and you get different vocabularies, even though the genetic code is nearly universal: the same triplets spell the same amino acids, the same stop signals end the genes. His own reaction was "what the hell". Yet a vocabulary built on the human genome trains fine on other species, with no strong drop in results.
The reason for the long context is just as concrete. A transcription factor can sit on a different chromosome from the gene it controls. If it mutates and stops working in one cell type, the gene it controlled breaks as well. A model can only learn that link if both stretches of DNA sit in the same context. Drop the repeats and filter out the junk, and a whole genome is still far longer than what today's chatbots take in.
Other groups read DNA without tokens at all. A 2024 paper in Science by Eric Nguyen and colleagues describes Evo, a seven-billion-parameter model trained on 2.7 million microbial and phage genomes, 131,000 letters at a time, one letter per step. It generated working synthetic CRISPR-Cas complexes.
So the same task is being attacked with ten-letter words and with single letters. Burtsev's open question sits underneath both: why would one genetic language need a different dictionary for every species?
