Chapter 0.8 — Architectural decision matrix

🎯 Chapter objective

Offer a practical instrument to decide between traditional ML, rules, workflow, simple LLM, RAG, fine-tuning, agent, multi-agent, or a hybrid solution.

🧠 Matrix

Question	Rules	Traditional ML	Simple LLM	RAG	Fine-tuning	Agent	Multi-agent	Hybrid (ML + LLM + agent)
Is the problem deterministic and well-defined?	✅	✅	⚠️	⚠️	⚠️	❌	❌	⚠️
Is there enough labeled data?	N/A	✅	⚠️	⚠️	✅	⚠️	⚠️	✅
Is the input open / ambiguous text?	❌	⚠️	✅	✅	⚠️	✅	✅	✅
Does the answer depend on mutable documents?	❌	⚠️	❌	✅	❌	✅	✅	✅
Is low latency required?	✅	✅	⚠️	⚠️	⚠️	❌	❌	⚠️
Must cost per task be minimal?	✅	✅	⚠️	⚠️	⚠️	❌	❌	⚠️
Must it act on external systems?	✅	⚠️	❌	❌	❌	✅	✅	✅
Is auditable explainability required?	✅	✅	⚠️	⚠️	⚠️	⚠️	⚠️	⚠️
Do relationships between entities matter?	⚠️	⚠️	⚠️	⚠️	❌	✅	✅	✅
Is there a strong regulatory requirement?	✅	✅	⚠️	⚠️	⚠️	⚠️	⚠️	⚠️
Can it involve real parallel decomposition?	❌	❌	❌	❌	❌	⚠️	✅	⚠️

Legend: ✅ = strong candidate; ⚠️ = feasible with care; ❌ = avoid.

🔭 Expanded matrix — operational dimensions

The matrix above answers "is this technically feasible?". The matrix below answers "is this operationally sustainable?".

Dimension	Rules	Traditional ML	Simple LLM	RAG	Fine-tuning	Agent	Multi-agent	Hybrid
Cost per call (relative)	Minimal	Low	Medium	Medium-high	Medium (after training)	High	Very high	Variable
Typical latency	<10 ms	<50 ms	0.5–3 s	1–5 s	0.5–3 s	5–60 s	10–300 s	Per step
Knowledge change frequency	Low	Low-medium	Static (training)	High (RAG indexes)	Low (needs retraining)	High via tools	High via tools	Variable
Need for style/behavior	No	No	Limited (prompt)	Limited	High	Variable	Variable	Variable
Need for external action	Direct	Limited	No	No	No	Direct	Direct	Direct
Regulatory risk	Low	Medium	High	High	High	Very high	Very high	Medium-high
Automated evaluation capability	High	High	Medium	Medium	Medium	Low	Very low	Medium
Required operational maturity	Low	High (MLOps)	Medium (LLMOps)	High (LLMOps+retrieval)	High (MLOps+evals)	Very high (AgentOps)	Extremely high	High
Rollback capability	Easy	Medium	Medium	Medium	Hard (weight version)	Hard (state, side-effects)	Very hard	Medium
Auditability	Total	High (model+features)	Partial	Partial (with citations)	Partial	Partial (semantic tracing)	Fragile	Variable

How to read it. The two matrices are complementary. A technically viable but operationally expensive decision is a recurring anti-pattern — agents in contexts where a classifier would solve the problem at a tenth of the cost, for example.

🧭 Rule of thumb

1. Start with the simplest. Rules > traditional ML > simple LLM > RAG > fine-tuning > agent > multi-agent.
2. Move up a rung only when the previous one does not solve it.
3. Hybrid is normal. ML for fast classification + LLM for semantic extraction + workflow + HITL is the most defensible enterprise pattern.

🧪 Architectural examples studied

• Classifier routing tasks to an LLM. A tfidf + linear classifier model decides in < 5 ms whether the case is "trivial" (deterministic answer), "documental" (RAG) or "action" (agent). It cuts LLM consumption by 40–80% in typical support flows.
• RAG with reranker and metadata filtering. Top-50 retrieval with BM25 + vector, mandatory filters by tenant_id and language, a cross-encoder reranker reducing to top-8. Without filters, the system surfaces documents from other customers; with a weak reranker, recall@8 drops 25–40%.
• Agent with tool calling under policy-as-code. A CRM-update agent exposes three tools (search, update, escalate). Every update call goes through a Rego policy validating tenant, role and allowed fields. High-risk update requires human approval.
• Fine-tuning for format/style. A model is fine-tuned to generate reports in the company's internal format (header, sections, abbreviations). Facts still come from RAG. Replacing factual knowledge with fine-tuning would be a recurring mistake.
• Distillation to reduce serving cost. A 70B model was distilled into a 7B model for a specific task (multi-label ticket classification), reducing inference cost ~10x while keeping F1 within an acceptable margin. The model is periodically re-evaluated against the teacher.

📌 Checklist

• [ ] Was the decision recorded in an ADR?
• [ ] Is it clear which problem each component solves?
• [ ] Was the added complexity justified?
• [ ] Who is the technical, product and operations owner?
• [ ] Which signals trigger a review of the decision?

---