MCP server testing needs two layers, because a server and the model using it fail in different places. A contract test on every pull request calls the server the way a client does, with no model, and pins the tool list, the arguments, the binding of each state handle to its caller, and the shape of an error. An agent eval runs on a schedule and when the pinned tool list changes, and checks that a model still picks the right tool and stops on a refusal.
Key facts
Definition
MCP server testing runs in two layers before an agent depends on the server: a contract test with no model, and an agent-in-the-loop eval.
How Bigcircle does it
- Bigcircle recommends running the contract test on every pull request, with no model in the loop, to pin the tool list, the arguments, the binding of each state handle to its caller, and the shape of an error.
- Bigcircle recommends running the agent eval on a schedule and again when the pinned tool list changes, because those are the moments a model can start using a correct server badly.
- On Solarpunk, Bigcircle built every tool as a typed contract with declared inputs, outputs and side effects, and a contract like that can be tested without a model.
Comparison
A demo shows that a model can call a tool once, while the contract test fails a pull request that leaks one caller's handle to another or turns a refusal into an empty list.
A demo where the model called a tool once tests neither. The server can drift, refuse badly or leak one caller's state to another while every demo stays green. A model on the happy path will not send a git ref (a branch or commit name) that starts with a dash, and it will not present a handle that another user's call created. A model can also stop using a tool well while the server is still correct.
The call returns, the trace looks finished, and neither the argument handling nor the handle binding was exercised. A contract test can show the server refused a foreign handle. It cannot show that a model will stop when it reads the refusal.
This guide covers what the contract pins and which official tools can run it. It then uses two published security bugs to shape the tests, and ends with what the agent eval adds and when each layer runs. Everything here is written against the 2026-07-28 MCP specification, which removed protocol sessions, and that change moves one of the pins.
Why does a tool contract need its own test?
On Solarpunk, the AI chief-of-staff agent we built, the first architecture wired each service to the model directly. Maintenance broke down in a specific way: a vendor changed a response and the break surfaced somewhere unrelated, days later. The replacement was one uniform tool layer, in which every tool is a typed contract with declared inputs, outputs and side effects.
A tool written that way can be tested without a model. The test speaks the protocol the way a client does, compares the reply with what the contract declares, and fails the pull request when the two differ. No prompt or score is involved, so nothing can be argued into a pass.
We drew the same line on AVOX, an insurance voice agent we built. Required disclosures live in the tool contract rather than the prompt. Whether the model will always make the disclosure is a question no test can answer. Whether the operation can run without the disclosure is one a test can answer, and on AVOX the answer is no. Most of what follows is a question of that second kind.
What does the contract pin on every pull request?
Start with the tool list. Keep a snapshot of the tools/list reply under version control, call the server through a client, and diff the reply against the snapshot: names, input schemas and descriptions. A description belongs in the snapshot because it is the instruction the model reads. Rewrite it and the agent follows the new text on its next run, while a pull request that pinned only the schema stays green.
As of the 2026-07-28 revision, the specification asks servers to return tools in a deterministic order, which keeps that diff stable. Every list result also carries a ttlMs and a cacheScope.
Pin the cache scope too, because a server that filters the list by the caller's scopes should mark it private. A public result may be cached by a shared gateway and served to another caller. MCP authorization covers why that leak matters at run time.
Arguments are the next pin, and they are where a tidy tool name can still hide a program. The specification makes input validation a requirement for every server. Require the fields the schema requires and reject a type the schema does not name. Reject, too, a value a downstream process would read as an option rather than as data. The access guide separates a declared tool from an escape hatch, and this test is how that separation survives the next edit.
Handle binding replaced session binding in this revision, because the Mcp-Session-Id header is gone. A server that needs state across calls returns a handle, such as a basket or workflow id, and the model passes it back as an ordinary argument.
Never let holding a handle count as authentication, which the specification's security guidance forbids. Bind each handle to the user taken from the verified token, for example by keying stored state on that user id and the handle, and reject a handle presented by anyone else.
The test creates state as one caller, presents the handle as a second caller, and requires a refusal that returns none of the first caller's data. It also presents a handle nobody issued and one that has expired. Both should return a tool execution error naming the problem, as the specification's non-normative guidance on stateful tools asks. A server that still speaks the 2025-11-25 revision to older clients needs the same test against its session ids.
The error shape is the last pin, and the specification separates two kinds of failure. Protocol errors, such as an unknown tool, come back as JSON-RPC errors. Tool execution errors come back as a result with isError set to true, so the model can read them.
Assert that a policy denial is the second kind, with the tool name and a reason code as structured output. Fail the build if a denied call comes back as an empty list with no error flag, because the model will narrate an empty list as missing data.
Which tools work for MCP server testing?
Split the work across three pieces of official tooling, because each checks something different. The versions below are their published releases as of 22 October 2026.
| Tool | What it gives the contract test | What it does not check |
|---|---|---|
TypeScript SDK, @modelcontextprotocol/client and /server 2.3.1 | The server you deploy, served in process through the client transport's fetch option, with auth info supplied for each request | Anything you do not write an assertion for |
MCP Inspector CLI, --cli mode | tools/list and tools/call from a shell, JSON output, and exit codes a pipeline can branch on | Whether the reply matches your snapshot, unless you diff it |
Conformance suite, @modelcontextprotocol/conformance | Protocol scenarios run against a live server URL: stateless requests, tool results and errors, caching, and checks on where a request came from | Your tools' own rules, such as a refund cap or a handle owner |
Build the contract on the SDK, because it lets one test present two different callers without a network. Pick its in-process path with care, since only one of the two covers the current revision. InMemoryTransport.createLinkedPair() links a client and a server in one process, but as of 22 October 2026 the SDK's own testing page limits it to instances speaking the 2025-era protocol. A handle test written on it exercises the older protocol, not the one 2026-07-28 clients speak.
For 2026-07-28, pass the deployed handler to the client transport as its fetch function, so every request is served in process by the same createMcpHandler you ship. The handler's second argument takes the verified auth info for that request, and the handle test needs exactly that. A sketch, with illustrative tool names and a server that reads the user from that auth info:
import assert from "node:assert/strict";
import { Client, StreamableHTTPClientTransport } from "@modelcontextprotocol/client";
import { createMcpHandler } from "@modelcontextprotocol/server";
import { createServer } from "../src/server";
const handler = createMcpHandler(createServer);
// One client per caller. Each request reaches the handler with the
// auth info your token verifier would have returned for that user.
async function clientAs(user: string) {
const authInfo = {
token: `test-${user}`,
clientId: user,
scopes: ["baskets"],
expiresAt: Math.floor(Date.now() / 1000) + 300,
};
const transport = new StreamableHTTPClientTransport(new URL("http://test.local/mcp"), {
fetch: (url, init) => handler.fetch(new Request(url, init), { authInfo }),
});
const client = new Client(
{ name: "contract-test", version: "1.0.0" },
{ versionNegotiation: { mode: "auto" } },
);
await client.connect(transport);
return client;
}
const alice = await clientAs("alice");
const bob = await clientAs("bob");
const created = await alice.callTool({ name: "create_basket", arguments: { sku: "A-100" } });
const { basket_id } = created.structuredContent as { basket_id: string };
const stolen = await bob.callTool({ name: "get_basket", arguments: { basket_id } });
assert.equal(stolen.isError, true);
assert.doesNotMatch(JSON.stringify(stolen), /A-100/);Assert on the whole reply. A server that refuses the call but echoes the basket in its error text has still leaked it, and a check on the error flag alone would pass it.
Use the Inspector's CLI mode as the quick check against a deployed server, not as the contract. It lists tools or calls one and exits, with --format json for a snapshot and exit codes a pipeline can branch on: 5 when a tool call returns isError or the tool is missing, 3 when the server wants authentication, and 4 when it cannot be reached. Add --stored-auth-only so it never opens a browser in CI. The diff against your snapshot is still yours to write.
Run the conformance suite for a different question: does this server speak the protocol correctly? Its scenarios for the 2026-07-28 revision ship in the 0.2.0 alpha releases, while the latest stable release, 0.1.16, still targets 2025-11-25. Pin the alpha if you want the stateless checks. Do not read a pass as covering your own rules, because the suite cannot know whether your refund tool refuses above its cap.
Where do MCP security tests belong?
Put security regressions on the pull request with the rest of the contract, for the same reason as an argument bug. An agent doing its task will not explore them. If the only test is a model completing a normal task, the bug ships, and the first caller who is not doing that task finds it.
Copy the argument test from a real fix. AWS security bulletin 2026-121-AWS, dated 1 October 2026, assigns CVE-2026-97662 to security-agent-mcp-server, an open-source server in the awslabs/mcp repository, for versions from 0.1.1 up to 0.2.0. The diff scan is open to argument injection. A crafted reference is read as a command-line option rather than a revision, and it can create, overwrite or truncate files outside the workspace.
The source builds the command as git diff, then the caller's ref, then a double dash. The separator is there, but it sits after the ref, so a value that starts with a dash is still parsed as an option. The 0.2.0 fix rejects an empty ref and a ref that starts with a dash before git runs. Its regression tests pass --not-a-ref and -o and assert that no subprocess was started.
Copy that shape: assert the refusal and assert that nothing was spawned. A test that only looks for a double dash in the argument list would have passed on the vulnerable command. The bad ref can also arrive as content in a codebase the agent was asked to scan. Prompt injection in multi-agent systems covers the check on the hop that carries content into an argument.
Take the binding test from MetaMCP, the open-source gateway from metatool-ai. In NVD's record for CVE-2026-79537, the session store through version 2.4.22 is keyed only by the client-supplied mcp-session-id header, with no owner, namespace or endpoint binding. A health endpoint also disclosed active session ids without authentication. A caller holding another tenant's id could list and run that tenant's tools, using the tenant's forwarded credentials.
The 2026-07-28 revision removed protocol sessions, and the same bug now has a new home. The specification's security guidance calls it state handle hijacking, and we treat the handle test above as the session test with a new name.
Refuse any launch whose command comes from the request body. CVE-2026-79538, in the same releases, is code execution through the gateway's inspector proxy endpoint for stdio servers, and CyberSecureToday's 4 October 2026 write-up traces it to a handler that takes process settings from the request rather than from a server record the caller owns. The project's GitHub releases still listed 2.4.22, from 19 December 2025, as the latest on 22 October 2026, so a gateway running it has no fix to install.
Neither of these tests needs a model. An agent signed in as one tenant, doing that tenant's work, will never present another tenant's id, so an eval stays green while the binding is missing.
What does an agent-in-the-loop eval add?
The agent eval asks what the contract cannot: given a task, does a model pick this tool, recover when the tool returns a typed error, and stop when the tool refuses? The specification tells clients to pass tool execution errors to the model so it can correct itself, and the eval is where you see whether it does. Those are properties of the model and the tool list together, so they move when either one moves.
Run the eval on a frozen set of tasks, the same cases each time. A failure is then a diff against the last green run rather than a new anecdote.
On the trademark drafting system we built, production logging compares the base and fine-tuned models on the same inputs. A regression from a model change shows up there as a score movement. The frozen task set does the same job for an MCP server when the model behind the agent changes.
Rubric design for that set is covered in how to build LLM evals. What belongs here is the cut between cases: tool choice, recovery from a typed error, and a stop on refusal. The run should reach only the server under test, because a general agent eval that can reach other systems will pass for reasons unrelated to this server.
MCP prompts versus tools for agent playbooks and how to design MCP tools decide what goes in the tool list. Once that shape is chosen, the agent eval shows whether a model can still follow it. A larger tool with a vague description will pass the schema test and fail the task set. You want to see that result before an agent in production depends on the server.
When does each layer run?
Each failure has a layer built to see it, and a layer that will stay green while the failure sits there. A run that tries to do both jobs ends up as a slow pull request that misses the argument bug, or a fast schema test that never notices the wrong tool.
| Failure | Contract test, every pull request | Agent eval, on a schedule or when the tool list changes |
|---|---|---|
| Tool list drift | The tools/list snapshot diff fails on a renamed field or a rewritten description | Sees the change only if a frozen task calls that tool |
| Argument handling | A value that would be read as an option is rejected before any process starts | A normal task never sends the bad value, so the run stays green |
| Handle binding | A handle presented by a caller it was not issued to is refused | One user's tasks pass. The other user's handle was never sent |
| Error shape | A denial is an error result with a reason code, not an empty list | Scores whether the model repeats the reason and stops |
| Tool choice | No model is in the run, so the test cannot see which tool would be picked | Scores the choice, the recovery from the typed error, and the stop |
Model calls stay off the pull request because they are slow, they vary between runs, and they miss the failures the contract exists for. A red agent eval blocks a merge for a reason the author cannot reproduce from the diff, and a green one lets an argument bug through.
Run the contract on every change to the server. Run the agent eval on a schedule, so a model or prompt change is caught even when the server did not move. Run it again when the pinned tool list in the pull request differs from the one on the main branch.
That trigger is a diff of the snapshot, not a judgement that the change might affect tools. A description edit counts even when the schema is identical, and a refactor that leaves the snapshot alone does not need a model.
When the agent eval fails, add a contract case if the server should never have produced that reply. Add a frozen task if the server replied correctly and the model used it badly.
A prompt change that hides a schema bug passes until the next model, and then the same bug is back. Keep the tool results in the eval run's observability so you can tell which fix you need without calling the model again.
AI agent monitoring covers the period after release, when real traces arrive and some look successful for the wrong reason. A server that has only been clicked through can still be the right one to build on. Other agents can depend on it once the contract has failed a bad change on a pull request and someone has read an eval run against its current tool list.
When is this more than you need?
A server with a few read-only tools, one user and no state across calls has nothing to bind and little to inject into. The tool-list snapshot and the error-shape test cover it. The scheduled agent eval can wait until a second team's agent depends on the server or a model change is planned.
Add the argument test the day a tool takes a value that reaches a shell, a query or a file path. Add the binding test the day a second user's state sits behind the same server.
Common questions
How do you test an MCP server?#
Test it in two layers. A contract test on every pull request calls the server through an SDK client with no model, diffs the tools/list reply against a snapshot, rejects bad arguments, refuses a handle presented by the wrong caller, and checks that a denial is an error result. An agent eval on a frozen task set runs on a schedule and when the snapshot changes, and scores tool choice and the stop on a refusal.
How do you check that a deployed MCP server works?#
Run the MCP Inspector in CLI mode against its URL. It lists the tools or calls one, prints JSON, and exits with a code a pipeline can branch on: 3 when the server wants authentication, 4 when it cannot be reached, and 5 when a tool call fails or the tool is missing. For protocol correctness, point the official conformance suite at the same URL.
Why not test the server by letting an agent call it?#
An agent on a normal task sends the arguments that complete the task. It does not send a ref that starts with a dash, and it does not present a handle that belongs to someone else. The call succeeds, and the two failures the contract exists for were never attempted. Use the agent eval for tool choice and for stopping on a refusal, and the contract for everything the server must do the same way on every call.
What should a contract test assert about arguments?#
Assert the schema, and assert the values a downstream process would misread. A missing required field fails, and so does a type the schema does not name. A ref that starts with a dash fails before any process is started. A separator placed after the value does not stop the process treating the value as an option. A test that only checks for the separator would have passed the vulnerable diff scan in CVE-2026-97662.
How do you test handle binding now that MCP has no sessions?#
Create state as one caller and present the returned handle as a second caller. The call must be refused without returning the first caller's data. Present a handle nobody issued and one that has expired, and expect a tool execution error that says so. The 2026-07-28 security guidance asks servers to bind each handle to the user from the verified token, and if any of these calls runs a tool, the contract failed.
Can the MCP Inspector run in CI?#
Yes. Its CLI mode, started with the cli flag, lists tools or calls one and exits, with JSON output and exit codes a pipeline can branch on. It exits 5 when a tool call returns an error or the tool is missing, 3 when the server needs authentication, and 4 when it cannot be reached. Use it for a quick check against a running server, and keep the snapshot diff and the binding tests in your own test code.
Does the official conformance suite replace a contract test?#
No. The conformance suite checks that a server speaks the protocol correctly, including stateless requests, tool results and errors, and caching. Its scenarios for the 2026-07-28 revision are in the 0.2.0 alpha releases, while the latest stable release, 0.1.16, targets 2025-11-25. It does not know your tools' rules, such as which user owns a handle or what a refund cap is, so run it beside your own contract test.
When should the agent-in-the-loop eval run?#
Run it on a schedule, so a model or prompt change is caught even when the server did not change. Run it again when the pinned tool list in the pull request differs from main, including a description edit with the same schema. Keep it off the other pull requests. A contract failure is enough to block those, and a model call on them adds time without covering the bug.
How is this different from production agent ops?#
Production ops watches traces after users are on the system, and it separates a bad answer from a failed call. This guide is the check before an agent is allowed to depend on the server. The contract fails the change that broke the server, and the agent eval fails the change that left the server correct but made the model use it badly. Once both have passed and real traffic arrives, the ops split takes over.
Further reading
- Model Context Protocol, Key changes in 2026-07-28. Sessions, the initialize handshake and
Mcp-Session-Idremoved; deterministic tool order; cache hints on list results. - Model Context Protocol, Tools. Input validation, stateful tools and handles, and the split between protocol errors and tool execution errors.
- Model Context Protocol, Security best practices. State handle hijacking and how to bind a handle to its user.
- Model Context Protocol, Inspector CLI. Methods, JSON output and exit codes for CI.
- Model Context Protocol, conformance and the TypeScript SDK, with its testing guide. Protocol scenarios against a server URL, and the in-process handler for tests.
- AWS Security Bulletin 2026-121-AWS, 1 October 2026, and NVD CVE-2026-97662. Argument injection in security-agent-mcp-server before 0.2.0.
- NVD, CVE-2026-79537 and CVE-2026-79538. MetaMCP through 2.4.22: session dispatch keyed only by the client-supplied header, and code execution through the inspector proxy.
- CyberSecureToday, MetaMCP unpatched RCE and session hijack, 4 October 2026. No patched release as of that date.
- MCP authorization. The token, the check before the call, and the refusal record that this contract pins.
- How to design MCP tools and MCP prompts versus tools for agent playbooks. What goes in the tool list before you pin it.
- How to build LLM evals. Rubrics and a frozen set for the agent-in-the-loop layer.
- AI agent monitoring. Quality and infrastructure once the server is a dependency.
- Solarpunk and AVOX. A typed tool layer that replaced point-to-point wiring, and compliance written into the tool contract.
- MCP, tool calling, evals, and guardrails.