MMOOZZGG.com

LLM

How do large language models work? From your question to the answer

9 min read12 sources

How do large language models work? A model like ChatGPT splits your question into tokens, turns them into numbers and writes the answer one token at a time, each time choosing a likely continuation. Almost all such models use the transformer architecture from a paper of 12 June 2017. Below are six steps on one diagram, the reasons for errors and a way to give the model your data.

On this page · 9 sections

Six steps on one diagram

An LLM (large language model) is a neural network trained to continue text. The path from your question to the answer fits into six steps, and the diagram below shows them in order.

The text becomes tokens. The tokens become numbers. Attention decides which words matter to each other. The model calculates the probabilities of the next token. The choice of token depends on two settings: temperature and top-p. Then it repeats until the answer ends.

The “next token” arrow leads from step 6 back to step 4: the chosen token is added to the text, and the model calculates the probabilities again, now taking it into account. This is how the answer grows word by word.

This scheme comes from the transformer architecture of the paper “Attention Is All You Need”: the network is built only on the attention mechanism, without the recurrent and convolutional layers of earlier models. What LLMs are in general and where they are used, see the section on language models.

Step 1. The text becomes tokens

The model does not read letters and words. First a tokenizer, a program that splits text, cuts the phrase into tokens: whole words, parts of words and symbols.

Sennrich and co-authors described splitting into word parts in 2015. A rare word is built from frequent pieces, so the model’s vocabulary is limited, yet it can write any word.

An example. On 16 October 2026 we checked the Russian phrase for “how a neural network works” in OpenAI’s tiktoken library. In the o200k_base encoding it took 5 tokens: the words for “how” and “works” stayed whole, and the word for “neural network” split into three pieces. In the cl100k_base encoding, which GPT-4 used to tokenize text, the same phrase took 9 tokens.

An English phrase with the same meaning took 6 tokens in both encodings. The lesson for the budget: the length of a question and an answer is counted in tokens, not words, and one phrase takes a different number of tokens in different models. More about this unit of payment in the glossary entry on tokens.

Step 2. The tokens become numbers

Each token has a number in the model’s vocabulary. By that number the model takes an embedding: a long list of numbers that describes the token. Words with close meanings get close lists.

Mikolov and co-authors tested the idea in 2013: word vectors trained on large texts capture how words are similar in meaning and in grammar. In a transformer, such vectors are learned together with the whole network.

Information about the token’s position in the phrase is added to the vector. Otherwise “the dog bites the postman” and “the postman bites the dog” would look the same to the model.

The same vectors also work in document search: the question and pieces of text are turned into numbers and compared. How this works is explained in the article on embeddings.

Step 3. Attention: which words matter to each other

The attention mechanism decides, for each token, which other tokens of the text to rely on and how strongly. In the phrase “The customer returned the item because it was defective”, attention links “it” to “item”, not to “customer”.

Bahdanau, Cho and Bengio proposed the idea in 2014 for machine translation: for each word of the translation, the model itself looks for the right words of the source phrase. In the transformer, attention became the basis of the whole network.

Attention is calculated in many layers in a row. Each layer refines the token vectors, and by the last layer the vector of the last token carries information about the whole phrase. More in the glossary: attention mechanism.

Step 4. Probabilities of the next token

From the vector of the last token, the model gets a score for every token in its vocabulary. The softmax function turns the scores into probabilities that add up to one.

For the question “Capital of France?” almost all the probability goes to the token “Paris”, and the other options get small shares. For the request “come up with a name for a coffee shop”, the probability is spread over hundreds of options.

Keep this in mind: the model does not look up the answer in a database of facts. It estimates which continuation is plausible after everything already written. This is both the model’s strength and the source of its errors.

Step 5. Choosing a token: temperature and top-p

The simplest way to choose is to always take the token with the highest probability. This is called greedy decoding, and in the Hugging Face Transformers library it works when random sampling is turned off.

In 2019 Holtzman and co-authors showed that if you always take the most likely continuation, the text comes out bland and strangely repetitive. So chatbots usually pick a token at random, but weighted by the probabilities.

Temperature changes the probabilities before the choice. A low temperature strengthens the likely tokens, and answers become predictable. A high one evens out the chances, and answers become more varied. In Transformers the default value is 1.0 unless the model sets its own.

Top-p, or nucleus sampling, cuts off the tail of unlikely tokens: what remains is the smallest set of options whose probabilities add up to at least p. At top-p 1.0 nothing is cut. To extract payment details from an invoice, set a low temperature; for advertising ideas, raise it.

Step 6. Repeat until the answer ends

The chosen token is added to the text, and the model again calculates the probabilities of the next token. An answer of a thousand tokens means a thousand such passes.

The loop stops when the model picks a special end-of-answer token or hits the limit on new tokens in the settings. That is why a chat answer prints piece by piece: the service shows tokens as they are chosen.

Everything the model sees at one time is limited by its context window: instructions, documents, the dialog history and its own answer. According to Anthropic’s documentation, Claude models have a window of 1 million tokens.

A long window does not mean exact reading. In 2023 Liu and co-authors found that models use information from the beginning and end of a long text more reliably and lose it more often in the middle.

Why the model makes mistakes and invents facts

Hallucination is a confident answer with an invented fact: a law article that does not exist, someone else’s figure, a link to nowhere. Steps 4 and 5 show the cause: the model writes a plausible continuation, and plausible is not always correct.

OpenAI researchers explain where this comes from in a paper of 4 September 2025. During pretraining, the model cannot tell a false statement from a fact by statistics, and the errors appear already there.

Tests then reinforce them: most of them score an answer as right or wrong, and a guess earns more points than an honest “I don’t know”. The authors propose changing how tests score answers, so that models do not learn to guess.

What to do in practice: give the model a source and ask it to cite it, set a low temperature for tasks with one right answer, and test the answers on your own set of questions before launch.

What the model does not know and how to give it your data

The model knows only what was in its training texts before its knowledge cutoff date. According to Anthropic’s documentation, the reliable knowledge of Claude models ends in June 2026. The model does not know about later events by itself.

The stated date is also approximate. In 2024 Cheng and co-authors showed that the real knowledge boundary often differs from the announced one: old pages and their copies end up in fresh sets of web texts.

Your rules, prices and contracts are not in the model at all. You give them through RAG (retrieval-augmented generation): the system finds the right pieces in your documents and passes them to the model with the question. In the 2020 work by Lewis and co-authors, answers with retrieval were more specific and more accurate on facts than those of a model without it.

If the documents hold customers’ personal data, Russian companies must follow 152-FZ, Russia’s personal data law: they cannot send such data to a cloud model abroad without meeting the law’s conditions. Then the model runs inside the company’s own perimeter, on its own server. More on law and data.

How such setups work in sales, support and document flow is covered on the page about LLMs for business.

Questions and answers

Does the model understand meaning or guess words?

The model predicts the next token, but to do that it builds numeric representations of meaning: embeddings and attention links between words. That is why it can translate, summarize and answer questions about a text. It has no fact check, though: it gives a plausible answer even when it does not know the right one.

Why does the same question get different answers?

At step 5 the model picks a token at random, weighted by the probabilities, and one different token sends the answer in another direction. Temperature and top-p set how much answers vary. For stable answers, set a low temperature and check the result over several runs.

How does the model know the latest news?

By itself, from nowhere: its knowledge ends at a cutoff date. Fresh information appears when the service connects web search or your documents and passes what it finds to the model with the question. Without search, the model answers about recent events with outdated information or makes something up.

Sources and fact-checking

Checked by the editors on 16 October 2026.

  1. arXiv, 12 June 2017 (arxiv.org)
  2. arXiv, OpenAI, 4 September 2025 (arxiv.org)
  3. arXiv, 31 August 2015 (arxiv.org)
  4. OpenAI tiktoken, GitHub, 16 October 2026 (github.com)
  5. arXiv, 16 January 2013 (arxiv.org)
  6. arXiv, 1 September 2014 (arxiv.org)
  7. arXiv, 22 April 2019 (arxiv.org)
  8. Hugging Face Transformers, 16 October 2026 (huggingface.co)
  9. arXiv, 6 July 2023 (arxiv.org)
  10. arXiv, 19 March 2024 (arxiv.org)
  11. Anthropic, Claude documentation, 16 October 2026 (platform.claude.com)
  12. arXiv, 22 May 2020 (arxiv.org)

Found a mistake? Tell us

Publications

Contact us