Skip to content

Chapter 2.1 — LLM fundamentals

🎯 Objective

Provide the minimum vocabulary and mental model of what exists inside an LLM to support decisions about architecture, cost and operation. This chapter does not teach how to train an LLM; it explains why reducing a prompt from 8k to 2k tokens changes more than the bill, or why two models of "the same size" can have radically different inference costs.

Each concept is presented with an operational definition, a practical consequence and, when applicable, a limit it imposes on architecture or budget.

🧠 Tokenization

A modern LLM does not see raw text: it operates on tokens, which are subword units learned by the tokenizer. The most common schemes are BPE (Byte Pair Encoding) and variants of SentencePiece (BBPE, Unigram).

Practical implications:

  • Tokens ≠ words. A word may become 1, 2 or more tokens, depending on the language and the model's vocabulary.
  • Portuguese and morphologically rich languages tend to generate more tokens per character than English. The same text costs more and takes up more context window in pt-BR.
  • Non-ASCII characters, emojis, source code and proper names produce irregular tokenization. Documents with a lot of code, JSON or XML have a token count that is hard to estimate off the top of your head.
  • Each provider has its own tokenizer. Counting tokens with model A's ruler to estimate cost on model B is a recurring source of FinOps surprises.

A typical architectural decision: measure the actual tokenizer of the target model over representative samples of the traffic before signing a contract or estimating window cost.

🧠 Internal embeddings vs retrieval embeddings

There are two kinds of "embedding" in the ecosystem, and confusing them causes problems in RAG.

  • Internal LLM embeddings. These are the vector representations the model uses internally, learned during training jointly with the network. They exist at the input layer and in the intermediate representations. They are not exported as a "retrieval embedding vector".
  • Retrieval embeddings. These are outputs of specific embedding models (for example text-embedding-3-small, bge-large, nomic-embed-text), trained so that vector similarity reflects semantic similarity between text passages.

Practical consequences:

  • The embedding model used in RAG is a separate decision from the generation LLM. Swapping the generator LLM does not invalidate the index; swapping the embedding model does invalidate the index and requires re-embedding.
  • "GPT-4's embedding" as a generic concept does not exist the way people often assume. For retrieval, use declared embedding models with a known dimension (see Ch. 2.8).

🧠 Transformer architecture (production view)

The Transformer (Vaswani et al., 2017) is composed of stacked blocks of:

  • Self-attention. For each position in the sequence, the model computes weights over all other positions and combines the representations. It costs O(n²) in compute and memory when done naively, where n is the number of tokens.
  • Feed-forward (MLP). Dense layers applied position-by-position.
  • Normalization and residuals. They stabilize training and inference.

What this means for architecture:

  • The larger the prompt, the more expensive the prefill. Not linearly: part of it grows with n² (naive attention). Modern variants (FlashAttention, sparse attention, sliding window) reduce the absolute cost, but the prompt × cost × latency trade-off still exists.
  • Models with the same "parameter count" can have very different inference costs because of the number of layers, number of heads, head dim, mixture-of-experts and attention strategy. "70B" is not a sufficient operational measure.
  • Long-context (>100k tokens) is done with techniques such as attention truncation, sparse/local attention and memory tokens. Each technique has its own cost and quality; a large window is not free, even when the provider does not explicitly charge for context tokens.

🧠 Context window

The context window is the maximum number of tokens the model accepts per call, prompt + response. It is a hard limit of the model, not a suggestion.

Consequences:

  • Everything that enters the context competes for window: instructions, documents (RAG), history, tool descriptions, tool results, internal plan, output contract.
  • Exceeding the window usually results in silent truncation: parts of the context disappear without an explicit warning. In RAG, this is a recurring source of "the model ignored the evidence".
  • "More window" does not replace context engineering (Ch. 2.3). Filling the context is often worse than selecting well.

🧠 Inference: prefill and decode

Inference in an autoregressive LLM has two phases with very different economics:

Phase What it does Limited by What it dominates
Prefill Processes the whole prompt in parallel, builds the KV cache Compute (FLOPs) TTFT (Time To First Token), prompt size
Decode Generates tokens one by one, reusing the KV cache Memory bandwidth TPOT (Time Per Output Token), response size

Therefore:

  • A large prompt tends to increase TTFT (latency to the first token) without necessarily affecting TPOT (latency per generated token).
  • A long response increases cumulative TPOT, even with a small prompt.
  • The wrong optimization goes to the wrong place. Reducing the prompt in a long-response scenario does not cure the perceived UX problem during decode.

This chapter is the basis for Ch. 6.3 (AI server sizing), which covers the quantitative reasoning.

🧠 KV cache

During decode, the model needs to read the keys and values (K/V) of the attention layers for all previous tokens. Recomputing this for every token would be prohibitive.

The solution is the KV cache: keys/values are computed in the prefill and stored, and decode only computes the new token.

Practical consequences:

  • The KV cache grows with the sequence length and with the number of simultaneous sequences. In production, it is often the largest VRAM consumer, even more than the weights themselves.
  • Techniques such as PagedAttention (vLLM) and continuous batching exist to manage this cache efficiently. Without them, large batching breaks due to lack of memory or fragmentation.
  • Long sequences have a hidden cost in VRAM, not just in billed tokens. In self-hosting, ignoring the KV cache is the most classic sizing mistake.

🧠 Sampling, temperature and limited determinism

The LLM output is a probability distribution over the next token. What comes out depends on the sampling strategy:

  • Greedy / temperature 0. Always takes the most probable token. Less creative, more predictable.
  • Temperature > 0. Smooths or sharpens the distribution before sampling.
  • Top-k. Restricts sampling to the k most probable tokens.
  • Top-p (nucleus). Restricts to the smallest set whose cumulative probability reaches p.
  • Repetition penalty, frequency/presence penalties. Adjustments to avoid pathological repetition.

An important and misunderstood detail:

Temperature 0 does not guarantee determinism. In production, the output still varies due to: non-deterministic order of GPU operations, dynamic batching, model version changes at the provider, hardware differences, and occasionally quantization bugs.

To reduce variation (not eliminate it) in production:

  • Pin model + version.
  • Whenever possible, the same provider and the same region.
  • Continuous eval to detect behavior drift.
  • Do not build production tests that assume identical strings; use semantic equivalence or contracts over structure.

🚨 Real limits of LLMs

Combining what appeared above, it becomes clear what an LLM is not:

  • It is not a source of truth. The training distribution is fixed, the model does not know the "now", and even in what it knows it can be confidently wrong.
  • It is not a dynamic knowledge base. Changing knowledge via fine-tuning is expensive, fragile and slow; RAG or tools change knowledge without retraining.
  • It is not symbolic reasoning. An LLM imitates patterns; "chain-of-thought" and related techniques help, but do not turn the model into a formal solver.
  • It is not an authorization mechanism. "Do not do X" in the prompt is an instruction, not a policy.
  • It is not deterministic. Even with temperature 0.
  • It is not stable across versions. Providers swap models; behaviors change without enough warning.

Each of these limits is addressed by some deterministic mechanism around the model (schema validation, RAG, policy-as-code, IAM, continuous eval). It is that engineering that makes the LLM operable.

🏢 Hermes Logística — wave 1.5

In the first LLM experiment, the Hermes team estimated cost using an English tokenizer over pt-BR tickets. The real bill, at the end of the first month, came in about 40% above the projection — with no change in traffic. Diagnosis: the actual tokenizer produced more tokens per ticket. Fix: measure with the target model's tokenizer over a representative sample before closing the next budget window. Lesson: token arithmetic only holds with the right tokenizer.

📚 References