What is an AI gateway? One governed endpoint between your apps or agents and the model providers they call. Every request passes through it, so that is where you route and fall back between models, keep provider keys out of agent code, and cap spend per team, user or agent. It is also where you run input and output guardrails, cache, and record a trace of each call. Bigcircle puts those controls on the request path of the agents it ships: credentials injected at the call, checks in code rather than in a prompt, and a trace that tells a planning mistake from an outage.
Key facts
Definition
An AI gateway is one governed endpoint for model traffic, where routing, fallbacks, provider keys, spend caps, guardrails, cache and a trace of each call all sit in front of the providers.
How Bigcircle does it
- On Solarpunk, Bigcircle injects each credential from an encrypted vault at execution time, so the model is called as a reasoning service and never holds the keys.
- On the trademark drafting system, Bigcircle logs the base and the fine-tuned model on the same inputs, so a routing or model change shows up as a score movement before traffic moves.
- Bigcircle measures each class of request on cost, latency and quality, then routes to the smallest model that still passes, with the routing table as configuration the client owns.
Comparison
An API gateway decides on a path and a method, while an AI gateway decides on tokens, prompts, streaming and model formats. An MCP gateway sits on tool traffic, not on model calls.
Key number
Woowa Brothers measured a prompt-cache hit rate moving from 5 percent to 97 percent on one project, and cost per request falling 84 percent, while average response time rose from 0.79 seconds to 1.33 seconds.
The problem starts as soon as a second team, a second agent or a second provider arrives. Each one carries its own key, its own bill and its own idea of which model to call. Finance sees one invoice, nobody can say which agent spent it, and there is no single place to stop the next call.
We met the credential half of that on Solarpunk, a desktop agent we built that reaches more than 100 tools. Credentials sit in an encrypted vault on the user's machine and are injected into the tool adapter at execution time, so the model is called as a reasoning service and never holds the keys. Direct wiring of every tool into the prompt held until roughly 30 tools. Past that, the model picked plausible tools from the wrong system with no sign in its output.
An AI gateway does the same job for model traffic: it holds the provider keys, picks the model, and writes the trace. It also concentrates the risk, because the box that holds every provider key is the first place an attacker looks. Ask where it keeps those keys before you ask how good its router is.
This guide starts with what an AI gateway is and how it differs from an API gateway. It then draws the line with an MCP gateway, lists what one should enforce for AI agents, and covers when you need one. It ends with a rollout order and how we place these controls on the agents we build.
The tool side of the same estate is what an MCP gateway does. Read that guide when the question is which MCP server an agent may call. Read this one when the question is which model, with which key, on whose budget.
What is an AI gateway?
Treat an AI gateway as a reverse proxy that understands tokens, prompts, streaming and model-specific request formats. Your app or agent sends every model call to one URL, and the gateway checks the caller's key, the remaining budget and the input against policy, picks a model, and forwards the request. If that provider fails it can fall back, then it writes a trace and returns the response, so every caller passes the same checks.
Walk one request through: an agent wants a completion, the gateway confirms the virtual key, refuses the call if the team or agent is over budget, and runs an input guardrail. It then routes to a model, or to a smaller model that still passes the quality floor, and if the chosen provider times out a fallback fires. The output guardrail runs, a cache may store the prompt for the next identical call, and one span lands in the trace with the agent, the model and the cost.
The figure shows the request path: blue marks the controls every model call shares, which belong in the AI gateway, and orange marks the tool path, which is a different door.
The hop is extra, and that is the trade: you pay a small wait and you own a new service so that keys, spend, policy and traces live in one place. Skip the gateway while one app talks to one provider on a bill you can already explain.
An AI gateway is also not the authorization of the action the agent takes after it gets the answer, because it governs the model call. Whether this person may refund this customer stays in the system of record, the same line MCP authorization draws for tools.
AI gateway vs API gateway
Keep your API gateway: TLS, authentication at the edge, rate limits by caller and request logs all carry over, along with the habits of a team that already runs one.
The unit of policy does not carry over. An API gateway decides on a path and a method, while an AI gateway has to decide on a model, a prompt, a token budget and a caller, for a request whose path often never changes.
| Concern | API gateway | AI gateway |
|---|---|---|
| Routing | By path and method | By model, caller and policy. The path is often one POST |
| Unit of cost | Requests or bandwidth | Tokens, cache writes and reasoning tokens, attributed per team, user or agent |
| Streaming | Optional | Completions stream. Timeouts have to fit tokens, not a short JSON body |
| Formats | One protocol per API | OpenAI, Anthropic and others differ. Some gateways translate. Some forward the native format |
| Guardrails | Headers, size, rate | Prompts and completions, including PII and topic policy |
| Caching | HTTP cache by URL | Prompt caching on a stable prefix. Volatile content in the prefix breaks the hit |
| Observability | Request logs | A generation span with model, tokens, cost, latency and the agent or key |
Streaming is the row that surprises people who treat the AI gateway as a thin proxy. A timeout that works for a short JSON response will cut a streamed completion in half, so set the timeouts for tokens, not for request count.
AI gateway vs MCP gateway
Keep the two doors apart, even when one product sells both. An AI gateway sits on model traffic: routing, fallbacks, provider keys, budgets, caching, guardrails on prompts and completions, and a trace of each generation. An MCP gateway sits on tool traffic: which server is approved, which tool this caller may run, which credential the call carries, and the audit line before a side effect.
Many products now do both. Kong AI Gateway 2.3, on its 4 November 2026 changelog, enforces policy on model calls as CallModel and on MCP traffic as CallTool. Snowflake's Cortex AI Gateway, generally available on 3 November 2026, is a model endpoint that accepts OpenAI Chat Completions and Anthropic messages formats, and its tools remain in preview. Name the hop you are buying. If the question is which MCP server an agent may reach, that work is already on the MCP gateway guide, and this page does not retarget those phrases.
A kill switch has the same split: cutting the AI gateway stops the next model response, and cutting the MCP gateway stops the next tool call. What is an AI agent kill switch is the runtime flag, and the gateway is one place that flag can sit.
What should an AI gateway enforce for AI agents?
Give the gateway seven jobs, and name an owner for each before the first agent is pointed at it. The table maps each job to a dated example from a product or a system we built. The product rows are each company's own account of what it offers on that date, so they show the control exists in the market and are not a test of it.
| Control | What the gateway does on each call | Dated example |
|---|---|---|
| Provider keys | Holds the upstream credential and attaches it on the gateway's side | Langfuse AI Gateway public beta, 4 November 2026: teammates get a gateway key, and the provider credential never leaves Langfuse. Solarpunk: a vault injects the secret at the call |
| Routing and fallbacks | Sends the request to a model that fits the class, and to a second model if the first fails | Snowflake Cortex AI Gateway, generally available 3 November 2026: one endpoint in OpenAI Chat Completions or Anthropic messages format. Bigcircle measures each class on cost, latency and quality |
| Spend caps | Limits tokens or dollars per team, user or agent, and refuses the next call at the cap | Snowflake: spend per user, team budgets, per-user quotas that block at a limit. MLflow AI Gateway, 3 November 2026 demo: Reject budget returns HTTP 429 |
| Guardrails | Checks the prompt and the completion in code before they pass | MLflow: PII and safety judges on the endpoint, HTTP 400 on a block. Insurance voice agent: the check sits in the contract, not in the prompt |
| Cache | Reuses a stable prompt prefix | Woowa Brothers, 6 November 2026, LiteLLM-based gateway: prompt-cache hit rate 5 percent to 97 percent on Project A after they moved volatile content out of the system prompt |
| Trace | Writes one span per call, with agent, model, tokens, cost and latency | Langfuse: each request recorded as a generation. Snowflake: spans in AGENT_TRACE_TABLE. Solarpunk: a trace per run |
| Disconnect | Turns off one agent, key or the whole model path without a deploy | Stacklok, 3 November 2026: administrator disconnect, refused with administratively_disconnected, reversible |
Keep provider keys in a vault the gateway reads at the call, never in agent code, environment variables or the prompt. On Solarpunk we inject the secret at the adapter. Kong Token Vault, on its 4 November 2026 account of AI Management in Konnect, keeps the agent from holding a token at all. Langfuse's public beta, the same week, hands each teammate a gateway key so the provider credential never leaves Langfuse.
Route by measuring, not by habit. Our model routing work splits traffic into classes, then measures each class on cost, latency and quality and sends it to the smallest model that still passes. The routing table is configuration the client owns, and a fallback covers a provider that degrades. Snowflake's Cortex AI Gateway, generally available 3 November 2026, accepts OpenAI Chat Completions and Anthropic messages at one endpoint. Langfuse's beta forwards the native format and does not translate between them, so keep the format your client already uses.
Cap spend on the request path. A dashboard that climbs after the fact has already paid for every call between the threshold and the person reading it, which is the argument on AI agent cost. Snowflake attributes spend per user, tracks team budgets, and blocks an individual at a per-user quota. MLflow's AI Gateway, in its 3 November 2026 write-up of a 3.17.1 demo, sets a Reject budget on the endpoint and answers later calls with HTTP 429. Put the check before the call, not after the consumer has gone over.
Write guardrails in code, because a prompt rule is a suggestion. MLflow runs PII and safety judges on the endpoint and returns HTTP 400 when they block. On an insurance voice agent we built, permission checks and required disclosures sit in the tool contract, so an operation cannot run without them. The same idea applies to the model hop: the gateway refuses a prompt or a completion that breaks policy, and the refusal is a typed error the agent can handle.
Cache only what is stable. Woowa Brothers, on their 6 November 2026 write-up of a LiteLLM-based AI gateway, moved volatile content out of the system prompt and set cache_control on Project A. On their measurements, the prompt-cache hit rate went from 5 percent to 97 percent and cost per request fell 84 percent, while average response time rose from 0.79 seconds to 1.33 seconds. They attribute the extra wait to LiteLLM looking up a Gemini cache on each request, which is why you measure latency when you turn caching on, not after the bill looks better.
Application-layer caching is a different saving: Prospex, our own research engine, runs cache-first so a rerun reuses extraction instead of calling external services again.
Join the gateway span to the agent trace. Langfuse records each gateway request as a generation, with the model, tokens, cost, latency and the key that called. Snowflake writes spans into AGENT_TRACE_TABLE, including models, timing, token counts and errors. On Solarpunk we keep a trace per run from the goal through each call to the final application state, and on Skionis one record links the request to every resulting resource.
AI agent monitoring depends on that join, because a refusal in the gateway log and an empty answer in the agent trace are the same event seen by two owners. Ship the line into the same observability stack.
Put a disconnect on the model path and test it. Stacklok's 3 November 2026 account of its kill switch has an administrator disconnect an AI gateway. In-flight calls are refused with administratively_disconnected, and turning it back on resumes traffic. That is the shape we want: a typed refusal, reversible, on the next request. Duplicate the halt at the gateway even if the runtime loop also reads it, so a client that bypasses your loop still hits the door.
Do you need an AI gateway?
Skip the gateway while one team runs one app against one provider, on spend you can already explain from the provider console. Put the key in a vault, pin the model id, and write a trace from the app. At that size a gateway is one more service on the request path, with an owner, an on-call rota and its own failure modes. It also adds a hop, so every completion waits on it.
Add one when any of these becomes true. A second team wants its own key, or a second client such as a coding agent joins the production app. Spend has to be attributed per team, user or agent, and the provider invoice cannot do it. You want a hard cap that stops the next call, or an audit of who sent which prompt. You are about to run a second provider, or a model you will swap under regression tests.
Each of those turns a decision made once per app into one you will otherwise make differently in each app. Count the triggers you already meet, and plan the gateway before the third client ships rather than after the bill arrives.
Be honest about the cost of the hop. Woowa's own measurements put extra latency on the cache lookup, and MLflow's demo write-up puts extra seconds on inline guardrail judges. Start in observe mode, which the next section covers, before you put a judge on every request. Run the gateway as production infrastructure from the first day it carries traffic, and alert on its refusal rate and its error rate separately, because a broken gateway that refuses everything looks like every agent refusing at once.
How to roll out an AI gateway
Inventory every model caller first: production apps, eval jobs, coding agents, notebooks and cron. A forgotten snapshot or a personal key will still bill after you think traffic has moved.
Point those callers at the gateway in observe mode. Log the model, the tokens, the cost and the caller, and do not enforce a budget or a guardrail yet. You need a week of your own mix before a cap is anything other than a guess.
Move the provider keys into the gateway vault and issue per-team or per-agent keys, then rotate the old keys. A gateway that still shares the installer's credential has not solved the original problem.
Add budgets next, as hard caps on the request path. Set each cap from the observe-mode traces, with room above the longest run that completed normally. Then turn on input and output guardrails, starting with the mechanical checks: schema, PII patterns, denied topics. Put a model judge on the path only after you have timed it.
Score a routing change the way we score a model swap. On the trademark drafting system, in production since 2023, we log the base model and the fine-tuned model on the same inputs against a 0 to 10 legal-integrity harness, and every draft passes that harness before an attorney sees it. A regression from a model change appears as a score movement.
Run the same pair on the current route and the new route, using the LLM evaluation cases you already have, before the new route takes traffic. We measured template errors before deploy falling from around 10% to 4% on Skionis once a validation loop checked every template. The number is about templates, not models, and the method is the same: measure the change against a held-out set before it reaches production.
How Bigcircle puts model calls on one path
We do not ship a product named AI gateway. We already put the jobs on the agents we build, and the design below moves in front of the providers without changing.
On Solarpunk we keep credentials in an encrypted vault and inject them at the call. The agent can act only through systems someone registered. We stopped giving the model the registry, and each step queries it, so the model is a reasoning service and never holds the keys.
On Skionis we treat the agent as untrusted. It acts only through purpose-built tools that wrap the provider's APIs, and there is no general shell. An engineer approves an architecture diagram before a change applies, and one record links the request to every resulting resource.
On the insurance voice agent, the model's output is a request to call a tool, and it is never itself an action. A request that fails escalates to a person and does not degrade into something close. On a current engagement for a telecom marketing platform, we scope each agent to the entitlements of the person it acts for. Prompts stay a user-experience layer that no policy check depends on.
The model routing work is the gateway's routing table: classes measured on cost, latency and quality, then sent to the smallest model that still passes, with fallbacks and a cache for repeated questions. The per-call policy is our AI guardrails work, and the span per call, with cost and latency on the step that spent them, is our AI observability work.
Across more than one agent or team, move those controls out of each runtime into one AI gateway in front of the providers. The vault, the route, the cap, the check and the trace then serve every caller the same way.
Common questions
What is an AI gateway?#
An AI gateway is one governed endpoint between your apps or agents and the model providers they call. Every request passes through it, so that is where you route and fall back, keep provider keys out of agent code, and cap spend per team, user or agent. It is also where you run input and output guardrails, cache, and write a trace of each call.
What is the difference between an AI gateway and an API gateway?#
Both handle authentication at the edge, rate limits and routing, but an API gateway decides on a path and a method. An AI gateway has to decide on a model, a prompt, a token budget and a caller. It also handles streaming completions, model-specific formats, prompt caching and guardrails on the content of prompts and completions.
Is an LLM gateway the same as an AI gateway?#
For this page, yes: teams use both names for the hop that sits on model traffic. An LLM gateway is the same idea under a narrower name: one endpoint for large language model calls, with keys, routing, budgets and traces. This page uses AI gateway because that is the phrase people type, and because some products on the hop also cover embeddings and other model classes.
Is there an open-source AI gateway?#
Yes. LiteLLM's documentation positions it as a self-hosted AI gateway: one OpenAI-compatible endpoint, virtual keys, spend tracking, fallbacks, guardrails and logs. MLflow's AI Gateway runs inside mlflow server and, on MLflow's own 3 November 2026 write-up, routes by endpoint name, runs PII and safety guardrails, applies a budget policy, and traces calls with agent, provider and endpoint tags. Langfuse's public beta can also run as an open-source gateway next to a self-hosted Langfuse. Pick by identity provider, hosting and the formats your clients already speak: there is no ranking here.
Does an AI gateway add latency?#
Yes. Every completion waits on an extra hop, and some controls add more. Woowa Brothers measured average response time rising from 0.79 seconds to 1.33 seconds after a cache lookup on each request, and MLflow's demo write-up puts extra seconds on inline guardrail judges. Start in observe mode, time your own path, and put a judge on every request only after you have that number.
Can an AI gateway stop a runaway agent?#
The gateway can stop the next model call, if the spend cap or the disconnect sits on the request path and is checked before the provider is called. A dashboard that emails later has already paid for every call in between. Cutting the AI gateway does not cancel a tool call that has already left, and it does not unwind a write the other system has already accepted.
Further reading
- Snowflake, Cortex AI Gateway inference, observability, and cost management (General availability), 3 November 2026. One governed endpoint, a trace per request, spend per user, team budgets, per-user quotas that block at a limit.
- Kong, Introducing AI Management in Kong Konnect, 4 November 2026. Cost management, Token Vault so agents never hold a credential, observability. Kong AI Gateway 2.3.0 changelog, same day: policy on model calls and on MCP tool calls.
- Langfuse, AI Gateway public beta, 4 November 2026. Native OpenAI and Anthropic requests, each call recorded as a generation, gateway keys so provider credentials are not shared.
- MLflow, Governing Coding-Agent Sprawl with MLflow AI Gateway, 3 November 2026. MLflow 3.17.1 demo: routing by endpoint name, PII and safety guardrails, budget policy, traces with agent, provider and endpoint tags.
- Stacklok, Why you need a kill switch for your MCP and AI gateways, 3 November 2026. Administrator disconnect,
administratively_disconnected, reversible. - LiteLLM, AI Gateway documentation, checked 9 November 2026. Self-hosted OpenAI-compatible endpoint, virtual keys, spend tracking, fallbacks, guardrails.
- Woowa Brothers, AI Gateway and AI Studio, 6 November 2026. LiteLLM-based gateway, Project A cache figures: 5 percent to 97 percent, 84 percent lower cost per request, average response time 0.79 seconds to 1.33 seconds. English summary: AI Briefing.
- What an MCP gateway does, AI agent kill switch, AI agent cost, AI agent monitoring, MCP authorization and how to build LLM evals.
- Model routing, guardrails, observability and prompt caching.
- Model routing, AI guardrails and AI observability.
- The Solarpunk build, the Skionis build, the insurance voice agent build and the trademark drafting build.