How a Model Actually “Thinks”

A language model receives tokens and calculates probabilities for what token could come next. A token may be a word piece, character group, byte sequence, or code fragment depending on the tokenizer. The model repeats the calculation after every generated token, building a response one step at a time.

It does not always choose the single most likely token. The surrounding decoder may use sampling controls such as temperature or top-p to choose among plausible candidates. That is why two runs from the same prompt can differ—and why lower-randomness settings can help with repeatable extraction while more varied settings can help brainstorming.

Calling this “autocomplete” describes the output interface, but not the full computation. A Transformer uses attention and many learned layers to build contextual representations of the input. Training at large scale, instruction tuning, tool use, and additional inference-time computation can produce planning, code generation, and problem-solving behavior that ordinary phone autocomplete never had.

A fluent answer is evidence that the text fits learned patterns—not that the claim was independently checked.

That distinction explains hallucinations. The generation process optimizes for a plausible continuation under its training and instructions. It does not automatically query a verified fact database before every sentence. A fabricated paper title can resemble real citations closely enough to be generated with confidence.

The material available for one response is the context window: your request, conversation content selected by the application, retrieved documents, tool results, and instructions such as a system prompt. When a conversation becomes too long, the product may drop, summarize, or compact earlier material. Behavior varies by product; old messages do not always disappear in one obvious step.

Product “memory” is separate. A chat service may save a profile, summary, or note and insert it into a later conversation. An agent may write facts into project files or a database. That is external state managed by the harness, not permanent learning inside the base model. The base model's weights do not normally change because you told one chat your preferred writing style.

Tools can add another layer. Search can provide current pages; retrieval can provide your documents; a calculator can check arithmetic. These tools can improve accuracy, but only if the right tool runs, returns good evidence, and the model represents that evidence correctly.

So the useful mental model is not “a database that talks” or “a person in a box.” It is a learned generator operating on selected context, sometimes with tools, producing a sequence that still needs verification when the stakes matter.