The vocabularyof shipped AI.
84 concepts that separate a demo from a production system, defined in plain language by engineers who have shipped them. 9 of them come with a diagram you can drive.
At a glance
Find a definition, read the explanation, and follow its sources or related terms when you need more detail.
- Who this is for
- Readers learning AI engineering terms or checking a concept during implementation.
- Topics
- LLM concepts
- Technologies
- Engineering practices
01
Models & Foundations
7- Foundation & Frontier ModelsConceptLarge pretrained models adaptable to many tasks; 'frontier' = the most capable current generation.
- Knowledge CutoffConceptThe date past which a model has no training knowledge, why RAG and tools exist.
- LLM Parameters & Model SizeConceptThe weight count (e.g. 8B, 70B) that roughly tracks capability, memory, and cost.
- Mixture-of-Experts (MoE)ConceptOnly a subset of the network ('experts') fires per token, so a huge model runs on a fraction of its compute.
- Pretraining vs Post-TrainingConceptPretraining builds raw knowledge; post-training (SFT, RLHF) makes it a usable assistant.
- Scaling LawsConceptLoss falls predictably with compute, data, and parameters, and the ratios matter as much as the totals.
- Transformer ArchitectureConceptThe neural-network architecture, built on attention, behind virtually every modern LLM.
02
How an LLM Works
8- Attention MechanismConceptInteractiveThe mechanism that lets each token weigh every other token when predicting the next.
- Context WindowConceptThe fixed token budget for prompt + output in a single call.
- LLM Temperature & SamplingConceptInteractiveTemperature, top-p, and top-k control how deterministic vs creative the output is.
- Logprobs & ConfidenceConceptPer-token log-probabilities expose model confidence and power low-cost eval signals.
- MQA, GQA & MLATechnologySharing keys and values across attention heads to shrink the KV cache, the quiet enabler of long context and lower-cost serving.
- Next-Token PredictionConceptInteractiveAn LLM is an autoregressive next-token predictor; everything else is sampling on top.
- RoPE & Positional EncodingConceptHow transformers know token order, and the rotary trick behind every modern long-context model.
- Tokens & TokenizationConceptInteractiveModels read and bill in tokens, subword chunks, not characters or words.
03
Prompting & In-Context Learning
7- Chain-of-Thought PromptingConceptAsking the model to reason step-by-step before answering improves hard tasks.
- Constrained DecodingTechnologyMasking invalid tokens at sampling time so output provably conforms to a schema, grammar, or regex.
- Few-Shot Prompting & In-Context LearningConceptModels learn a task from examples in the prompt, no weight updates.
- Prompt Engineering & System PromptsConceptStructuring instructions, roles, and context to reliably get the output you want.
- Prompt InjectionConceptUntrusted input hijacking the model's instructions, the top LLM security risk.
- Reasoning Models & Extended ThinkingConceptModels that spend extra inference compute 'thinking' before answering (test-time compute).
- Structured Output & Constrained DecodingConceptForcing the model to emit valid JSON/schema-conformant output every time.
04
Retrieval & RAG
10- AI Hallucination & GroundingConceptConfident, fluent, wrong, and how grounding + citations contain it.
- Chunking StrategiesConceptHow you split documents determines what retrieval can ever find.
- EmbeddingsConceptInteractiveText mapped to vectors so semantic similarity becomes geometric distance.
- GraphRAGTechnologyBuild a knowledge graph from your corpus and retrieve through entities and relationships instead of isolated chunks.
- Hybrid SearchConceptCombine keyword (BM25) and vector search to catch what each misses.
- Late Interaction & ColBERTTechnologyKeep one vector per token instead of one per passage, and match query to document token-by-token at search time.
- RerankingConceptA second-stage cross-encoder reorders top-k results for precision.
- Retrieval Metrics (Recall@k, MRR, NDCG)PracticeWhether the right passage arrived and how high it ranked are separate questions with separate fixes.
- Retrieval-Augmented Generation (RAG)ConceptFetch relevant documents at query time and feed them into the prompt as grounding.
- Vector Search & Vector DatabasesTechnologyApproximate nearest-neighbor search over embeddings, the retrieval engine of RAG.
05
Agents & Tool Use
14- Agent MemoryConceptHow agents carry state across steps and sessions beyond the context window.
- AI AgentConceptInteractiveAn LLM in a loop that decides actions toward a goal using tools and feedback.
- AI Software FactoryPracticeA shared, governed pipeline where coding agents build, test and verify code from a spec, so people review outcomes.
- CodeActConceptAgents that act by writing executable code instead of emitting one JSON tool call at a time.
- Computer Use Agents (GUI Agents)ConceptAgents that drive real interfaces with screenshots, clicks, and keystrokes instead of APIs.
- Decision ModelConceptA model that picks from a list the harness wrote and returns a probability, used inside an AI agent for the small repeated choices.
- Loop EngineeringPracticeDesigning the checks, event triggers and trace-driven harness changes around an AI agent, rather than tuning one prompt.
- Meta HarnessPracticeAn outer loop that searches over harness code around a fixed model, or a looser name for a layer that runs several coding-agent harnesses.
- Model Context Protocol (MCP)TechnologyAn open standard for connecting models to tools and data sources.
- Multi-Agent OrchestrationConceptSplitting work across specialized agents coordinated by a supervisor or shared plan.
- Plan-and-ExecuteConceptDraft the whole plan up front, execute steps with a smaller model, and replan only when reality disagrees.
- ReAct Agents (Reason + Act)ConceptInterleave reasoning traces with tool actions so the model plans as it acts.
- Reflexion & Self-CorrectionConceptAgents that critique their own output and retry, turning failures into a feedback signal.
- Tool Calling (Function Calling)ConceptThe model emits a structured call to a function you defined, and you run it.
06
Inference & Serving
10- Continuous BatchingTechnologySwap finished sequences out and new ones in every step instead of waiting for the slowest.
- Disaggregated ServingTechnologyRun prefill and decode on separate GPU pools and ship the KV cache between them, so the two phases stop fighting.
- FlashAttentionTechnologyAn IO-aware exact-attention kernel that tiles the computation in on-chip SRAM, making long context feasible.
- KV CacheConceptCaching attention keys/values so each new token doesn't recompute the whole sequence.
- LLM QuantizationConceptStoring weights in fewer bits (e.g. 4-bit) to shrink memory and speed inference.
- PagedAttentionTechnologyInteractiveVirtual-memory-style paging of the KV cache to cut fragmentation and fit more sequences.
- Prefill vs DecodeConceptInteractiveTwo structurally different phases: parallel prompt prefill, sequential token decode.
- Speculative DecodingConceptA small draft model proposes tokens a big model verifies in parallel, cutting latency.
- Tensor & Pipeline ParallelismTechnologyHow a model bigger than one GPU runs: split each layer across GPUs, split layers into stages, or both.
- Time to First Token vs Time Between TokensPracticeTime-to-first-token and time-between-tokens budget differently and must be measured apart.
07
Cost & Latency Levers
3- Batch APITechnologySubmit large non-urgent workloads asynchronously for a steep per-token discount.
- Model Routing & CascadingPracticeSend easy requests to a small model and escalate only hard ones to a frontier model.
- Prompt CachingPracticeReuse the model's work on a repeated prompt prefix to cut cost and TTFT dramatically.
08
Fine-Tuning & Adaptation
10- Catastrophic ForgettingConceptFine-tuning on a narrow set can erase general capabilities the base model had.
- Constitutional AI & RLAIFConceptAlign a model against a written set of principles by having it critique and revise its own outputs, then train on the result.
- DPO (Direct Preference Optimization)ConceptPreference tuning without a separate reward model or RL loop, simpler than RLHF.
- GRPOTechnologyPPO without the value model: sample a group of answers, score them against each other, push toward the better ones.
- LoRA & QLoRA Fine-TuningTechnologyTrain tiny adapter matrices instead of all weights, fast, low-cost, swappable fine-tuning.
- Model DistillationConceptTrain a small, low-cost student to mimic a large teacher model on your task.
- RLHFConceptReinforcement learning from human preference data, how base models become helpful assistants.
- RLVR (Verifiable Rewards)ConceptReinforcement learning where the reward is a checkable fact, tests pass or the answer matches, instead of a learned preference.
- Supervised Fine-Tuning (SFT)ConceptContinuing training on curated examples to specialize a model's behavior.
- Synthetic DataPracticeTraining on model-generated examples, the workaround for the data wall and the fastest way to build SFT sets.
09
Evaluation & Safety
10- Benchmark ContaminationConceptWhen test sets leak into training data, scores measure memory instead of capability.
- Data PoisoningConceptPlanting malicious examples in training or retrieval data to backdoor a model's behavior.
- Evals & LLM Evaluation MetricsPracticeTask-specific test suites that tell you if a prompt/model change helped or hurt.
- Groundedness (Faithfulness)PracticeWhether every factual claim in an answer is supported by the passages that were retrieved.
- Judge Alignment (TPR, TNR, Cohen's Kappa)PracticeRaw agreement hides a judge that passes everything, so alignment needs separate error rates.
- LLM GuardrailsTechnologyInput/output filters that block unsafe, off-topic, or policy-violating content.
- LLM Observability & TracingPracticeSpan-based traces of every prompt, tool call, and token so you can debug non-determinism.
- LLM Red Teaming & JailbreaksPracticeAdversarially probing the model to find failures before users (or attackers) do.
- LLM-as-a-JudgeConceptUsing a strong model to grade outputs at scale, powerful but bias-prone.
- Mechanistic InterpretabilityConceptReverse-engineering the circuits and features inside a model instead of treating it as a black box.
10
Multimodal & Voice
5- Diffusion ModelsConceptInteractiveImage generators that start from noise and iteratively denoise toward a prompt.
- Realtime Voice AgentsConceptFull-duplex spoken conversation: listen, think, and speak with human-like turn-taking.
- Speech-to-Text (ASR)TechnologyTranscribing audio to text, Whisper and streaming ASR for voice apps.
- Text-to-Speech (TTS)TechnologyNeural voices that synthesize natural speech, streamable for low-latency agents.
- Vision-Language Models (VLMs)ConceptModels that take images + text together, OCR, charts, screenshots, document understanding.
A term you cannot explain is a system you cannot debug.
9 of them you can take apart.
- AI AgentAn LLM in a loop that decides actions toward a goal using tools and feedback.
- Attention MechanismThe mechanism that lets each token weigh every other token when predicting the next.
- Diffusion ModelsImage generators that start from noise and iteratively denoise toward a prompt.
- EmbeddingsText mapped to vectors so semantic similarity becomes geometric distance.
- LLM Temperature & SamplingTemperature, top-p, and top-k control how deterministic vs creative the output is.
- Next-Token PredictionAn LLM is an autoregressive next-token predictor; everything else is sampling on top.
- PagedAttentionVirtual-memory-style paging of the KV cache to cut fragmentation and fit more sequences.
- Prefill vs DecodeTwo structurally different phases: parallel prompt prefill, sequential token decode.
- Tokens & TokenizationModels read and bill in tokens, subword chunks, not characters or words.
Let's build something that ships.
Tell us what you're building. We'll tell you whether it needs a fixed-scope build, a prototype taken to production, or a programme we run, what's achievable, and where the real bottlenecks are.