The vocabularyof shipped AI.
80 concepts that separate a demo from a production system, defined in plain language by engineers who have shipped them. 9 of them come with a diagram you can drive.
At a glance
Find a definition, read the explanation, and follow its sources or related terms when you need more detail.
- Who this is for
- Readers learning AI engineering terms or checking a concept during implementation.
- Topics
- LLM concepts
- Technologies
- Engineering practices
01
Models & Foundations
7- Foundation & Frontier ModelsConceptLarge pretrained models adaptable to many tasks; 'frontier' = the most capable current generation.
- Knowledge CutoffConceptThe date past which a model has no training knowledge, why RAG and tools exist.
- Mixture-of-Experts (MoE)ConceptOnly a subset of the network ('experts') fires per token, so a huge model runs cheaply.
- Parameters & Model SizeConceptThe weight count (e.g. 8B, 70B) that roughly tracks capability, memory, and cost.
- Pretraining vs Post-TrainingConceptPretraining builds raw knowledge; post-training (SFT, RLHF) makes it a usable assistant.
- Scaling LawsConceptLoss falls predictably with compute, data, and parameters, and the ratios matter as much as the totals.
- TransformerConceptThe neural-network architecture, built on attention, behind virtually every modern LLM.
02
How an LLM Works
8- AttentionConceptInteractiveThe mechanism that lets each token weigh every other token when predicting the next.
- Context WindowConceptThe fixed token budget for prompt + output in a single call.
- Logprobs & ConfidenceConceptPer-token log-probabilities expose model confidence and power cheap eval signals.
- MQA, GQA & MLATechnologySharing keys and values across attention heads to shrink the KV cache, the quiet enabler of long context and cheap serving.
- Next-Token PredictionConceptInteractiveAn LLM is an autoregressive next-token predictor; everything else is sampling on top.
- RoPE & Positional EncodingConceptHow transformers know token order, and the rotary trick behind every modern long-context model.
- Temperature & SamplingConceptInteractiveTemperature, top-p, and top-k control how deterministic vs creative the output is.
- Tokens & TokenizationConceptInteractiveModels read and bill in tokens, subword chunks, not characters or words.
03
Prompting & In-Context Learning
7- Chain-of-ThoughtConceptAsking the model to reason step-by-step before answering improves hard tasks.
- Constrained DecodingTechnologyMasking invalid tokens at sampling time so output provably conforms to a schema, grammar, or regex.
- Few-Shot & In-Context LearningConceptModels learn a task from examples in the prompt, no weight updates.
- Prompt Engineering & System PromptsConceptStructuring instructions, roles, and context to reliably get the output you want.
- Prompt InjectionConceptUntrusted input hijacking the model's instructions, the top LLM security risk.
- Reasoning & Extended ThinkingConceptModels that spend extra inference compute 'thinking' before answering (test-time compute).
- Structured Output & Constrained DecodingConceptForcing the model to emit valid JSON/schema-conformant output every time.
04
Retrieval & RAG
10- ChunkingConceptHow you split documents determines what retrieval can ever find.
- EmbeddingsConceptInteractiveText mapped to vectors so semantic similarity becomes geometric distance.
- GraphRAGTechnologyBuild a knowledge graph from your corpus and retrieve through entities and relationships instead of isolated chunks.
- Hallucination & GroundingConceptConfident, fluent, wrong, and how grounding + citations contain it.
- Hybrid SearchConceptCombine keyword (BM25) and vector search to catch what each misses.
- Late Interaction & ColBERTTechnologyKeep one vector per token instead of one per passage, and match query to document token-by-token at search time.
- RerankingConceptA second-stage cross-encoder reorders top-k results for precision.
- Retrieval Metrics (Recall@k, MRR, NDCG)PracticeWhether the right passage arrived and how high it ranked are separate questions with separate fixes.
- Retrieval-Augmented Generation (RAG)ConceptFetch relevant documents at query time and feed them into the prompt as grounding.
- Vector Search & Vector DatabasesTechnologyApproximate nearest-neighbor search over embeddings, the retrieval engine of RAG.
05
Agents & Tool Use
10- Agent MemoryConceptHow agents carry state across steps and sessions beyond the context window.
- AI AgentConceptInteractiveAn LLM in a loop that decides actions toward a goal using tools and feedback.
- CodeActConceptAgents that act by writing executable code instead of emitting one JSON tool call at a time.
- Computer Use & GUI AgentsConceptAgents that drive real interfaces with screenshots, clicks, and keystrokes instead of APIs.
- Model Context Protocol (MCP)TechnologyAn open standard for connecting models to tools and data sources.
- Multi-Agent OrchestrationConceptSplitting work across specialized agents coordinated by a supervisor or shared plan.
- Plan-and-ExecuteConceptDraft the whole plan up front, execute steps cheaply, and replan only when reality disagrees.
- ReAct (Reason + Act)ConceptInterleave reasoning traces with tool actions so the model plans as it acts.
- Reflexion & Self-CorrectionConceptAgents that critique their own output and retry, turning failures into a feedback signal.
- Tool / Function CallingConceptThe model emits a structured call to a function you defined, and you run it.
06
Inference & Serving
10- Continuous BatchingTechnologySwap finished sequences out and new ones in every step instead of waiting for the slowest.
- Disaggregated ServingTechnologyRun prefill and decode on separate GPU pools and ship the KV cache between them, so the two phases stop fighting.
- FlashAttentionTechnologyAn IO-aware exact-attention kernel that tiles the computation in on-chip SRAM, making long context feasible.
- KV CacheConceptCaching attention keys/values so each new token doesn't recompute the whole sequence.
- PagedAttentionTechnologyInteractiveVirtual-memory-style paging of the KV cache to cut fragmentation and fit more sequences.
- Prefill vs DecodeConceptInteractiveTwo structurally different phases: parallel prompt prefill, sequential token decode.
- QuantizationConceptStoring weights in fewer bits (e.g. 4-bit) to shrink memory and speed inference.
- Speculative DecodingConceptA small draft model proposes tokens a big model verifies in parallel, cutting latency.
- Tensor & Pipeline ParallelismTechnologyHow a model bigger than one GPU runs: split each layer across GPUs, split layers into stages, or both.
- TTFT vs TBT (Latency Metrics)PracticeTime-to-first-token and time-between-tokens budget differently and must be measured apart.
07
Cost & Latency Levers
3- Batch APITechnologySubmit large non-urgent workloads asynchronously for a steep per-token discount.
- Model Routing & CascadingPracticeSend easy requests to a cheap model and escalate only hard ones to a frontier model.
- Prompt CachingPracticeReuse the model's work on a repeated prompt prefix to cut cost and TTFT dramatically.
08
Fine-Tuning & Adaptation
10- Catastrophic ForgettingConceptFine-tuning on a narrow set can erase general capabilities the base model had.
- Constitutional AI & RLAIFConceptAlign a model against a written set of principles by having it critique and revise its own outputs, then train on the result.
- DistillationConceptTrain a small, cheap student to mimic a large teacher model on your task.
- DPO (Direct Preference Optimization)ConceptPreference tuning without a separate reward model or RL loop, simpler than RLHF.
- Fine-Tuning (SFT)ConceptContinuing training on curated examples to specialize a model's behavior.
- GRPOTechnologyPPO without the value model: sample a group of answers, score them against each other, push toward the better ones.
- LoRA & QLoRATechnologyTrain tiny adapter matrices instead of all weights, cheap, fast, swappable fine-tuning.
- RLHFConceptReinforcement learning from human preference data, how base models become helpful assistants.
- RLVR (Verifiable Rewards)ConceptReinforcement learning where the reward is a checkable fact, tests pass or the answer matches, instead of a learned preference.
- Synthetic Training DataPracticeTraining on model-generated examples, the workaround for the data wall and the fastest way to build SFT sets.
09
Evaluation & Safety
10- Benchmark ContaminationConceptWhen test sets leak into training data, scores measure memory instead of capability.
- Data PoisoningConceptPlanting malicious examples in training or retrieval data to backdoor a model's behavior.
- EvalsPracticeTask-specific test suites that tell you if a prompt/model change helped or hurt.
- Groundedness (Faithfulness)PracticeWhether every factual claim in an answer is supported by the passages that were retrieved.
- GuardrailsTechnologyInput/output filters that block unsafe, off-topic, or policy-violating content.
- Judge Alignment (TPR, TNR, Kappa)PracticeRaw agreement hides a judge that passes everything, so alignment needs separate error rates.
- LLM Observability & TracingPracticeSpan-based traces of every prompt, tool call, and token so you can debug non-determinism.
- LLM-as-a-JudgeConceptUsing a strong model to grade outputs at scale, powerful but bias-prone.
- Mechanistic InterpretabilityConceptReverse-engineering the circuits and features inside a model instead of treating it as a black box.
- Red-Teaming & JailbreaksPracticeAdversarially probing the model to find failures before users (or attackers) do.
10
Multimodal & Voice
5- Diffusion ModelsConceptInteractiveImage generators that start from noise and iteratively denoise toward a prompt.
- Realtime Voice AgentsConceptFull-duplex spoken conversation: listen, think, and speak with human-like turn-taking.
- Speech-to-Text (ASR)TechnologyTranscribing audio to text, Whisper and streaming ASR for voice apps.
- Text-to-Speech (TTS)TechnologyNeural voices that synthesize natural speech, streamable for low-latency agents.
- Vision-Language Models (VLMs)ConceptModels that take images + text together, OCR, charts, screenshots, document understanding.
A term you cannot explain is a system you cannot debug.
9 of them you can take apart.
- AI AgentAn LLM in a loop that decides actions toward a goal using tools and feedback.
- AttentionThe mechanism that lets each token weigh every other token when predicting the next.
- Diffusion ModelsImage generators that start from noise and iteratively denoise toward a prompt.
- EmbeddingsText mapped to vectors so semantic similarity becomes geometric distance.
- Next-Token PredictionAn LLM is an autoregressive next-token predictor; everything else is sampling on top.
- PagedAttentionVirtual-memory-style paging of the KV cache to cut fragmentation and fit more sequences.
- Prefill vs DecodeTwo structurally different phases: parallel prompt prefill, sequential token decode.
- Temperature & SamplingTemperature, top-p, and top-k control how deterministic vs creative the output is.
- Tokens & TokenizationModels read and bill in tokens, subword chunks, not characters or words.
Let's build something that ships.
Tell us what you're building. We'll tell you whether you need an engineer embedded or the whole build led, what's achievable, and where the real bottlenecks are.