Serving and inference

Managed API endpoints from providers like OpenAI, Anthropic, or Together are the right default: they abstract away GPU procurement, scaling, and model updates, and the per-token price is usually cheaper than self-hosting until you reach sustained, high-utilization workloads. The math flips when you have volume that keeps GPUs saturated around the clock, when you need a model not offered by any provider, or when data-residency and control requirements rule out third-party processing. Start with an API, measure actual usage, and revisit the calculation when you have real numbers.

Sources

• Together AI inference overviewTogether AI
• Amazon Bedrock overviewAWS

The true cost of self-hosting is not the per-token GPU rate but the amortized hourly cost of the instance divided by the tokens you actually produce each hour at your real utilization level. A GPU running at 20% utilization is five times more expensive per token than its spec sheet suggests. Build the model before deciding: take your monthly token volume, divide it across the hours in a month, convert that into required GPU-hours, price them, and compare directly against the API invoice you would receive for the same tokens.

Sources

• How continuous batching enables throughputAnyscale

Sending prompts and completions through a managed API means your data travels to and is processed on infrastructure you do not control, which creates compliance risk for regulated industries and sensitive workloads. Self-hosting puts both the weights and every inference request inside your own perimeter, satisfying HIPAA, GDPR, and internal data-governance policies that prohibit third-party processing. Data residency is one of the two non-negotiable reasons to self-host; the other is needing a model no provider offers.

Sources

• vLLM documentationvLLM

Managed inference platforms that host open and frontier models behind an API, removing the need to operate GPU infrastructure. Amazon Bedrock integrates with AWS IAM and VPC controls and supports models from Anthropic, Meta, Mistral, and others. Together AI and Fireworks AI specialize in open-weight models with low-latency endpoints, per-token pricing, and dedicated deployment options. All three handle provisioning, scaling, and model updates, making them the practical starting point before evaluating self-hosting.

Sources

• Amazon Bedrock overviewAWS
• Together AI quickstartTogether AI
• Fireworks AI introductionFireworks AI

vLLM is the most widely adopted open-source LLM serving engine, developed at UC Berkeley. It introduced PagedAttention for near-zero KV cache waste and continuous batching to keep GPUs saturated, and it supports over 200 model architectures, quantization formats from FP8 to GGUF, and tensor and pipeline parallelism across multiple GPUs. When you decide to self-host, vLLM is the default starting point before considering more specialized alternatives.

Sources

• vLLM documentationvLLM
• Efficient Memory Management for LLM Serving with PagedAttentionarXiv

Unless you already know you have a data-residency requirement or a volume that makes self-hosting cheaper, an API is almost always the faster and more economical path. The hidden costs of self-hosting, including GPU reserved instances, on-call burden, model update cycles, and the engineering time to tune batching and memory, are large and easy to underestimate. Ship with an API, instrument your actual token volume and latency requirements, and only revisit the decision when evidence changes the math.

Sources

• Together AI quickstartTogether AI

A GPU that sits idle waiting for traffic is pure waste. Self-hosting becomes cost-competitive only when the GPU runs at high utilization consistently, typically above 50 to 70 percent, because the denominator in cost-per-token is the tokens produced per hour, not the tokens the card could produce per hour. Model your real traffic distribution across the day, account for idle time overnight or on weekends, and price the instance accordingly. Bursty workloads with long quiet periods almost always favor managed APIs.

Sources

• How continuous batching enables throughputAnyscale

Traditional static batching holds a fixed group of requests together until every sequence in the batch has finished generating, which means a short response early in the batch wastes GPU cycles waiting for a long one to complete. Continuous batching, also called in-flight or iteration-level batching, inserts new requests into the batch the moment a slot opens, keeping the GPU close to full utilization at all times. The Anyscale benchmark showed vLLM achieving up to 23x throughput improvement over naive static batching using this technique paired with PagedAttention.

Sources

• How continuous batching enables 23x throughput improvementAnyscale
• Efficient Memory Management for LLM Serving with PagedAttentionarXiv

PagedAttention treats the KV cache the way an OS treats RAM: it divides cache memory into fixed-size pages and stores each sequence's attention keys and values in non-contiguous blocks, allocating and freeing them on demand. Before PagedAttention, serving engines pre-allocated a contiguous chunk of GPU memory for each request's maximum possible context length, leaving most of it empty for short sequences and making it impossible to fit many requests at once. By eliminating that internal and external fragmentation, PagedAttention enables near-zero KV cache waste and makes larger batch sizes possible without increasing GPU memory.

Sources

• Efficient Memory Management for LLM Serving with PagedAttentionarXiv
• vLLM documentationvLLM

During autoregressive decoding, the model must attend over every previous token in the context. Rather than recomputing the key and value projections for those tokens at every step, the engine caches them in GPU HBM. This KV cache is the primary consumer of GPU memory during serving: for a long-context request, the KV cache can dwarf the model weights themselves, and it grows with both sequence length and batch size. Understanding how the KV cache is managed, whether through PagedAttention, prefix caching, or quantized KV, is central to understanding why batch size and context length are constrained the way they are.

Sources

• Efficient Memory Management for LLM Serving with PagedAttentionarXiv
• How continuous batching enables throughputAnyscale

vLLM is the open-source default for high-throughput LLM serving, combining PagedAttention for memory efficiency and continuous batching for GPU utilization. It exposes an OpenAI-compatible REST API, supports over 200 model architectures, and integrates quantization formats including AWQ, GPTQ, FP8, and GGUF. For most teams starting self-hosted serving, vLLM is the first engine to reach for before tuning toward more specialized alternatives.

Sources

• vLLM documentationvLLM
• Efficient Memory Management for LLM Serving with PagedAttentionarXiv

TensorRT-LLM is NVIDIA's compiled inference library for running LLMs at the lowest possible latency on NVIDIA hardware. It uses ahead-of-time compilation to fuse kernels, prune operations, and generate GPU-specific code optimized for the target architecture, whether Ampere, Hopper, or Blackwell. The tradeoff for this latency advantage is operational complexity: models must be compiled before deployment, and the compiled engine is specific to the hardware and precision it was built for. It is the right choice for latency-critical production deployments on NVIDIA where vLLM's interpreted overhead is measurable.

Sources

• TensorRT-LLM GitHubNVIDIA

SGLang is a high-performance serving framework built around RadixAttention, an efficient prefix-caching mechanism that reuses KV cache across requests sharing a common prefix, such as a long system prompt. It also features a zero-overhead CPU scheduler and tight integration with structured output generation, making it especially efficient for agentic and structured-output workloads where prompt prefixes repeat across requests. SGLang is deployed at scale by xAI, LinkedIn, and major cloud providers.

Sources

• SGLang: Efficient Execution of Structured Language Model ProgramsarXiv
• SGLang GitHubSGLang

Text Generation Inference (TGI) is Hugging Face's production serving toolkit, supporting continuous batching, tensor parallelism, and quantization for popular open-weight models. It has entered maintenance mode as Hugging Face now recommends vLLM or SGLang for new deployments. Ollama is designed for the opposite end of the spectrum: local and developer use, bundling model management and a lightweight server that runs quantized models on consumer hardware via llama.cpp, making it the standard way to test an open-weight model on a laptop before committing to a production inference stack.

Sources

• Text Generation Inference documentationHugging Face
• Ollama documentationOllama

No single engine is optimal for every workload. Batch processing jobs that maximize tokens per dollar favor vLLM with large continuous batches and high-throughput settings. Latency-bound interactive chat may benefit from TensorRT-LLM's compiled kernels or SGLang's efficient scheduling. Structured-output-heavy agentic pipelines see outsized gains from SGLang's RadixAttention when prompts share long prefixes. Local development and prototyping are best served by Ollama. Choosing the right engine for the workload, and tuning its settings, is often more impactful than hardware upgrades.

Sources

• vLLM documentationvLLM
• SGLang GitHubSGLang

Continuous batching lifts throughput by keeping the GPU full, but large batches can increase the latency of individual requests, because new tokens for one sequence cannot be emitted until the engine finishes an iteration across all sequences in the batch. Sarathi-Serve's chunked-prefill technique addresses this by splitting large prefill requests into equal-sized chunks processed across multiple iterations, preventing a long prefill from monopolizing the GPU and blowing out tail latency for concurrent decode-phase requests. Tune batch size and chunk size jointly against your actual TTFT and TBT SLOs rather than maximizing throughput in isolation.

Sources

• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-ServearXiv
• How continuous batching enables throughputAnyscale

LLM inference has two structurally different phases. Prefill processes the entire input prompt in one forward pass: because all prompt tokens are available simultaneously, computation is highly parallel and the phase is compute-bound, saturating the GPU's FLOP capacity. Decode generates output one token per autoregressive step: each step reads the full model's weights and the KV cache but writes only a single new token, so the phase is memory-bandwidth-bound rather than compute-bound. The practical implication is that a long prompt can be processed quickly in prefill, but generating a long output is fundamentally limited by how fast the GPU can stream bytes from HBM.

Sources

• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-ServearXiv
• How continuous batching enables throughputAnyscale

Time to first token (TTFT) measures the wall-clock time from request arrival to the first generated token and is dominated by the prefill phase plus any queuing delay. Time between tokens (TBT), also called inter-token latency, measures the gap between successive output tokens and is bounded by decode-phase memory bandwidth. These are independent metrics with different root causes, different optimization levers, and different user experience impact: TTFT governs perceived responsiveness, while TBT governs streaming smoothness. Tracking only a single blended latency number hides which phase is the actual bottleneck.

Sources

• Prometheus monitoring overviewPrometheus

Serving throughput (tokens per second across all requests) and per-request latency pull in opposite directions. Packing more requests into a batch improves GPU utilization and raises aggregate throughput, but each request must wait for a full batch iteration to complete before its next token is emitted, which raises TBT and can stall requests that arrived mid-batch. The optimal operating point depends on your service-level objectives: a batch pipeline optimizes for throughput; a latency-sensitive chat UI may need smaller batches or strict queue-admission limits. There is no universally correct batch size.

Sources

• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-ServearXiv

GPU memory must accommodate three things: the model weights, the activations for the current batch, and the KV cache for all in-flight requests. For a 7B-parameter model in FP16, weights consume roughly 14 GB. On a 40 GB A100 that leaves about 26 GB for KV cache, and how many requests fit depends on sequence length: at 4K tokens per request, KV cache per sequence for a typical transformer is several gigabytes, so only a handful of concurrent sequences fit. Quantizing the KV cache or the weights directly increases the budget for concurrent requests. Doing this arithmetic before deployment tells you your real batch-size ceiling before you ever run a load test.

Sources

• vLLM documentationvLLM
• Efficient Memory Management for LLM Serving with PagedAttentionarXiv

Prometheus scrapes numeric metrics from your inference engine on a configurable interval and stores them as time-series data; Grafana queries that data to render dashboards and fire alerts. For LLM serving the critical metrics to expose and track are TTFT at the 50th and 99th percentiles, TBT at the same percentiles, tokens per second per replica, GPU memory utilization, batch size distribution, and queue depth. vLLM and TGI both expose a Prometheus-compatible metrics endpoint out of the box, so the integration requires only a scrape config and a dashboard.

Sources

• Prometheus overviewPrometheus
• Getting started with GrafanaGrafana

Because TTFT and TBT arise from different phases of inference and respond to different optimization levers, they need separate SLO targets and separate alerting thresholds. A degraded TTFT most often points to queue depth, prefill overload, or a cold-start problem; a degraded TBT points to decode-phase memory bandwidth saturation or too-large a batch. Blending them into a single end-to-end latency metric hides which phase broke and delays diagnosis. Define p99 budgets for each independently and page on whichever threshold trips.

Sources

• Prometheus overviewPrometheus

Before running a load test, do the arithmetic: for your target model, compute the per-token KV cache size (2 x num_layers x num_heads x head_dim x bytes_per_element), multiply by your maximum sequence length and desired concurrent-request count, and check whether it fits in the remaining GPU memory after weights. In practice the KV cache, not the model weights, is what caps batch size and maximum context. Quantizing the KV cache to INT8 roughly halves this budget requirement, and PagedAttention ensures the allocated budget is not wasted on internal fragmentation.

Sources

• Efficient Memory Management for LLM Serving with PagedAttentionarXiv
• vLLM documentationvLLM

Quantization reduces the bit-width of model weights, and sometimes activations, from the training-time precision (BF16 or FP16) to lower-precision formats such as INT8, INT4, or FP8. Fewer bits per parameter means smaller model size, lower GPU memory pressure, and faster memory-bandwidth-bound decode, all of which translate directly into larger batch sizes and higher throughput at the same hardware cost. GPTQ and AWQ are the dominant post-training quantization approaches for 4-bit weight-only quantization; FP8 is hardware-accelerated on Hopper GPUs; GGUF is the quantized format used by llama.cpp for CPU and edge inference. Always evaluate quality on your own task after quantizing, because accuracy loss is real and varies by model and format.

Sources

• AWQ: Activation-aware Weight QuantizationarXiv
• GPTQ: Accurate Post-Training QuantizationarXiv
• vLLM quantization documentationvLLM

Speculative decoding uses a small, fast draft model to propose several candidate tokens at once, which a larger target model then verifies in a single parallel forward pass. Because the target model can accept or reject each draft token in parallel rather than generating one at a time, correct drafts are free; only rejections pay the full per-token cost. The result is 2 to 3x latency reduction for the target model with no change to its output distribution, provided the draft model is reasonably aligned. The technique works best when output tokens are predictable, as in coding or templated responses, where draft acceptance rates are high.

Sources

• Fast Inference from Transformers via Speculative DecodingarXiv
• vLLM speculative decoding documentationvLLM

When a model does not fit on a single GPU, it must be split across multiple devices. Tensor parallelism shards individual weight matrices horizontally across GPUs within a node, so each GPU holds a slice of every layer; this is the preferred strategy for single-node multi-GPU deployments because the all-reduce communication stays on fast NVLink. Pipeline parallelism instead assigns consecutive layers to different GPUs or nodes, so stage boundaries require only point-to-point activation transfers; this is necessary for multi-node deployments and is typically combined with tensor parallelism, setting tensor-parallel degree to GPUs per node and pipeline-parallel degree to the number of nodes.

Sources

• vLLM parallelism and scalingvLLM

AWQ (Activation-aware Weight Quantization) and GPTQ are the two dominant methods for post-training 4-bit weight quantization of large language models. GPTQ uses approximate second-order information to minimize the reconstruction error of each layer's weights under quantization, enabling accurate 3- to 4-bit quantization with a one-time calibration pass. AWQ takes a different approach: it identifies a small fraction of weights that are salient based on activation magnitudes and protects them from quantization, achieving comparable accuracy with lower calibration cost and better hardware utilization. Both formats are supported natively by vLLM and TensorRT-LLM for production serving.

Sources

• AWQ: Activation-aware Weight QuantizationarXiv
• GPTQ: Accurate Post-Training QuantizationarXiv

GGUF is the binary model format used by llama.cpp, a C/C++ inference library that targets CPU and edge hardware. A GGUF file bundles quantized weights, tokenizer vocabulary, and model metadata into a single self-contained file that can be loaded on a laptop, a Raspberry Pi, or any machine without a GPU. Quantization levels from Q2_K to Q8_0 let you trade model quality for memory and speed on constrained hardware. Ollama wraps llama.cpp with a model registry and HTTP server to make local serving accessible; the GGUF ecosystem is the standard path for edge inference and developer experimentation before committing to a GPU-backed stack.

Sources

• llama.cpp GitHubllama.cpp
• Ollama documentationOllama

Quantization reliably reduces memory footprint and raises throughput, but quality loss is real and varies unpredictably across models, tasks, and quantization formats. AWQ and FP8 tend to be the most accuracy-preserving approaches; aggressive 4-bit and lower formats can degrade performance on reasoning-heavy or knowledge-intensive tasks. The correct workflow is: quantize, then run your full evaluation suite on the quantized model before promoting it to production. A quantized model that passes your evals is safe to deploy; one that fails needs a less aggressive format or a different calibration approach.

Sources

• AWQ: Activation-aware Weight QuantizationarXiv
• vLLM quantization documentationvLLM

The first request to a fresh replica incurs the full cost of loading the model from storage to GPU memory: for a 70B model this can take 30 to 60 seconds, which is unacceptable as user-visible latency. The standard mitigations are keeping at least one replica always warm, pre-warming replicas before predicted traffic spikes using scheduled scaling, and loading weights from fast instance-local NVMe or from memory-mapped model stores rather than object storage on every cold start. If your platform scales to zero, make cold-start latency explicit in your SLO so it is accounted for in product design rather than discovered by users.

Sources

• vLLM documentationvLLM