In the last two years, the context window arms race has been relentless. Claude's flagship models moved from 200,000 tokens to a full 1 million token context window in 2026. Gemini pushed further, with some models reaching 2 million tokens, and Google has claimed figures as high as 10 million for its largest model. GPT's line settled around the same 1 million token mark for its top tier.
On paper, that's the equivalent of feeding a model several long novels in a single request. It's tempting to read that number and conclude memory is solved. If a model can hold that much text at once, surely it can just remember everything.
It can't. And the reason why is one of the more important, least understood distinctions in how these systems actually work.
A context window is not storage
A context window is the amount of text a model can look at during a single request. It is not a place where information lives between requests. Close the conversation, start a new one, and every token of that million-token window is gone. There is no persistence, no memory, nothing carried forward unless you (or some system sitting between you and the model) explicitly puts it back in.
Think of a context window less like a filing cabinet and more like a very large desk. You can spread a huge number of documents across it. But the moment you leave the room, the desk is cleared. Coming back means laying every document out again, by hand, from scratch.
That's the first gap. Size describes capacity, not retention.
Size doesn't mean the model uses it evenly
Even within a single request, while all that text is genuinely sitting in the window, models don't treat every part of it equally. This has been studied directly, and the results are consistent across nearly every major model tested.
The foundational paper here is Lost in the Middle: How Language Models Use Long Contexts (Liu et al., Stanford and UNC, Transactions of the Association for Computational Linguistics, 2024). It tested models on tasks where they had to find a specific piece of information buried somewhere in a long input. The finding, since replicated in follow-up work like "Found in the Middle":
- Performance is highest when the relevant information sits at the very beginning or very end of the context.
- Performance drops significantly when that same information sits in the middle, even in models explicitly marketed as long-context.
- This produces what researchers describe as a U-shaped attention bias. The model reliably favors the start and end of what it's given, and reliably neglects the middle, regardless of how important the buried content actually is.
A newer and, in some ways, more sobering study extends this further. NoLiMa (short for "No Literal Matching," Adobe Research, 2025) tested 13 popular LLMs that all claim support for context windows of at least 128,000 tokens. Instead of testing simple keyword lookup, NoLiMa removed literal overlap between the question and the answer, forcing models to reason about the meaning of what they'd read rather than pattern match on shared words.
At 32,000 tokens, 11 of the 13 models dropped below 50% of their own short-context accuracy.
Even one of the strongest performers in the test, GPT-4o, fell from a near-perfect 99.3% baseline at short context down to 69.7% once the context grew long and literal matches were removed. Later analysis in the same line of research found that models' effective context length, the point where they still retain at least 85% of their short-context accuracy, is often a small fraction of their advertised maximum. One estimate put GPT-4.1's effective context length at around 16,000 tokens, despite a stated capacity in the hundreds of thousands.
Independent industry benchmarking backs this up from a different angle. NVIDIA's RULER benchmark estimates that usable context typically sits at 50 to 65% of the advertised window size across popular models, not because the tokens vanish, but because reasoning quality over that much material degrades well before the window technically fills up.
So what does a bigger window actually buy you?
Real things. Just not the thing "memory" implies.
- More material in a single pass. You can paste in a longer document, a bigger codebase, a longer transcript, and have the model consider it in one shot.
- Fewer round trips. Less need to manually chunk and re-feed information across multiple calls.
- Better performance on some long-document tasks, genuinely, when the needed information happens to sit near the start or end of the input, or when literal keyword matching does most of the work.
What it does not buy you:
- Persistence. Nothing carries over to your next session automatically.
- Uniform attention. The model doesn't treat token 500,000 with the same care as token 500.
- Reasoning that holds up at scale. The NoLiMa and Lost in the Middle results both point the same direction. As context grows, especially past the tens of thousands of tokens, the model's effective understanding of what's buried inside it degrades, often sharply.
Memory is a different engineering problem
Real memory, the kind that lets a system say "you told me about this three weeks ago, in a different tool, in a different conversation," isn't solved by handing a model a bigger desk. It requires deciding, deliberately:
- What's worth keeping, out of everything that was ever said. Not everything is worth remembering. Most conversation is genuinely disposable.
- How it's stored so it can be found again later, independent of any single conversation's context window.
- How it's retrieved, at the right moment, without dumping every past interaction back into the window and re-triggering the exact "lost in the middle" problem described above.
That's retrieval, extraction, and judgment. A different set of engineering problems from "how many tokens can I fit." A model with a 2 million token window and zero memory architecture will still forget everything the instant the conversation ends. A model with a modest window paired with a real memory system can recall something from a month ago instantly, because it was never relying on the window to hold it in the first place.
The industry's context windows will keep growing. That trend isn't slowing down. But treating window size as a proxy for memory is a category error the research has been quietly correcting for two years now. These are related but genuinely different problems, and conflating them is exactly how "just use a model with a bigger context window" becomes advice that quietly stops working the moment you actually test it.
