A context window is not durable memory; it is closer to the workbench laid out for one generation. A wider bench can hold more material, but placing the right sentence on it does not guarantee that the model will use it correctly. Conversation history, RAG, and long-term memory are separate processes that store information, select it, and place it back on that bench.
Editor's note — what this means for enterprise buyers — "A one-million-token context" does not mean you can simply load every internal document and be done. This part separates parameters, conversation history, retrieval, and long-term memory. Use it when deciding how to connect internal knowledge, and how that choice lands on cost and access control.
A continuous chat is still assembled as a new request
A chat interface lets us scroll naturally through a conversation that began yesterday. It feels as though the model remembers the exchange, but in an ordinary call the model receives the bundle of messages that the application sends again with the current request.
If an earlier turn is included, the model can see it. If it has been dropped or replaced with a summary, the original is unavailable. A record being visible in the interface and present in the model's current input are different states.
This distinction matters when something goes wrong. When a model appears to forget an agreed condition, the first check is not its memory capacity but whether the condition remains in the actual request payload. A model with a long context cannot use information that was not sent.
Parameters, context, and memory live in different places
Around an LLM, the word memory is used for several different things. Without a distinction, “the model knows it” and “the model read it in this request” get mixed into the same sentence.
| Where information lives | When it is created | How it changes | Main role |
|---|---|---|---|
| Model parameters | Pre-training and post-training | Further training or weight replacement | Broad patterns of language and code |
| Current context | Every request | Prompt and attachment assembly | Working information for this generation |
| Conversation or work memory | Between requests | Application storage and updates | Preferences, decisions, progress state |
| Retrieval store | Document ingestion and indexing | Document and index updates | Source material and current evidence candidates |
Parameters retain patterns learned from data, but are not an original-document repository. Context can supply an original passage directly, but it is bounded by tokens and one request. Memory and retrieval sit outside the model, so separate rules decide what to store and when to bring it back.
Once these locations are separated, the update path becomes clearer. When a company policy changes, updating the retrieval store may be faster than retraining a model. When a user requests a format for one report only, the current prompt is a better home than long-term memory.
Context length is a shared budget for input and output
Context length is normally stated in tokens. The budget contains more than the user's question. System instructions, earlier messages, attached documents, tool calls and results, and room for the new answer all contribute.
total context budget
= system instructions
+ conversation history
+ retrieved or attached material
+ tool requests and observations
+ the answer to be generated
Adding documents gives the model more information to inspect but can leave less room for the answer. Long input also increases processing time and KV-cache memory. The exact growth depends on architecture and serving, but a context window is not free storage.
What gets removed at the limit is an application decision. Mechanically dropping the oldest turn can remove the goal agreed at the beginning. Keeping every turn can let duplicated experiments occupy more space than the evidence that matters.
Information present is not necessarily information used
A sufficient context limit lets an application insert a long document in full. “Can be supplied” and “can be used reliably” still need to be tested separately.
The Lost in the Middle study moved relevant information to different positions in multi-document question answering and key-value retrieval tasks. In the models evaluated, performance was often substantially lower when the relevant material was in the middle of a long input than at the beginning or end. A long-context specification did not mean information at every position was used equally well.
One study does not stand in for every current model and task. It does leave a useful criterion: context evaluation needs more than maximum length. Relevant evidence should be moved to different positions, surrounded with more irrelevant material, and tested in tasks that require several pieces of evidence to be connected.
In practice, we use clear sections and headings rather than placing a critical instruction once in the middle of a long passage. Important conditions can be restated near the question. Reducing material is not only a cost optimisation; it helps define where the model should allocate attention.
RAG puts retrieved material back into the context
RAG stands for retrieval-augmented generation. Instead of relying only on model parameters, it retrieves relevant passages from external documents and supplies them in the current context before generation.
The early RAG paper combined parametric memory in a pre-trained model with non-parametric memory in a Wikipedia vector index for knowledge-intensive tasks. The important shift was to supplement weights with external knowledge that could be updated and traced to a source.
A common RAG flow can be simplified to four stages.
formulate the question
→ retrieve and rank relevant documents
→ place selected passages in context
→ generate an answer using the evidence
RAG does not change model parameters. It adds reference material to this request. Updating the document store can therefore affect the next retrieval, and the system can retain which documents were supplied with an answer.
Adding retrieval divides failure into four locations
RAG supplies evidence but does not automatically establish factuality. The query can miss the intent, the correct document can be absent from the index, chunking can separate the needed sentence, or the model can produce a conclusion unsupported by the retrieved passage.
Looking only at the final answer hides which component needs work.
| Stage | Question to check |
|---|---|
| Ingestion and indexing | Is the needed document present and current? |
| Retrieval and ranking | Does the evidence appear near the top? |
| Context assembly | Is the needed sentence intact and clearly separated? |
| Generation | Does each important claim actually agree with its evidence? |
If retrieval is accurate but the answer is wrong, generation instructions or citation validation need attention. If the correct document never enters the request, changing the model comes after fixing the index and retriever. RAG becomes explainable only when retrieval and answer quality are measured separately.
Long-term memory also needs rules for what not to keep
Conversation memory stores earlier information and inserts it into a later request. A small state can retain user preferences, decisions already made, and completed steps while using fewer tokens than the entire conversation.
Summarisation is compression and deletion at the same time. A memory such as “the customer prefers short reports” can lose the condition that this preference applied only to one project. If a model inference is stored as confirmed fact, a mistaken premise can become more persistent in later conversations.
Memory benefits from provenance, scope, and an update policy. The system can distinguish what the user stated from what the model inferred, whether it applies to one task or indefinitely, and whether new information replaces or versions the old value.
Information lifetime gives a practical storage rule.
needed for this answer only → current prompt
needed during one task → task state
reused across conversations → validated long-term memory
original source and freshness key → document store + RAG
change broad model behaviour → post-training
The next part will look more closely at how an answer is formed inside that context. We will examine when longer reasoning helps, how to distinguish plausible reasoning from supported conclusions, and why evaluation needs to change with the task.
If you are evaluating adoption
Three points decide how connecting internal knowledge actually goes.
First, what you select matters more than how much you can fit. A wider context lets you load more material, but if retrieval pulls the wrong document, that document becomes the basis for the answer. Retrieval quality generally sets the ceiling on answer quality.
Second, permissions must not leak through the context. In an organisation where people see different documents, access control has to apply at the retrieval step. Instructing a model to "see this but do not mention it" is not a control.
Third, do not contract for memory when what you have only looks like memory. Conversation history, a search index, and long-term memory differ in where data is stored, how long it is kept, and who is responsible for deleting it. For work involving personal or customer data, that distinction is a security requirement.
Next → Does longer reasoning make an answer better? (Part 4)