Skip to content

Part 2 — Summary

Part 2 treated the LLM as a probabilistic component within a larger architecture. The points that were established:

  • Fundamentals matter for production decisions, not for training a model from scratch. Tokenizer, KV cache, prefill vs decode and sampling determine cost, latency and variance.
  • Quantization and compression trade precision for capacity. The gain is only real when the kernel supports the format on the target hardware — measure, do not assume.
  • Context engineering is more important than prompt engineering. Selecting, ordering and formatting what enters the context weighs more than trying to squeeze out performance with a better instruction.
  • Structured outputs guarantee form, not content. Validation remains the application's responsibility.
  • AI gateways are operational governance layers, not security mechanisms nor substitutes for evals.
  • RAG is the dominant path for corporate knowledge, but it only works when filters, reranker, chunking, freshness and per-tenant isolation are treated as engineering, not as a detail.
  • Embeddings and Matryoshka turn embedding dimension into an architectural decision with direct effects on cost, RAM and quality.
  • LLM evaluation is its own system, with versioned datasets, golden + adversarial + regression, and human review where applicable.

Part 3 goes up a rung: agents take action in the world, and everything discussed about the LLM starts operating within an autonomous loop with tools, memory and state.