How to Test MCP Tool Schemas in CI

How to Test MCP Tool Schemas in CI

A tool's contract with the world is its schema, not its code. As long as tools/list returns the same name, inputSchema, and outputSchema, you can rewrite everything behind a tool and every client keeps working. Change the schema — even a field that seems harmless — and clients can break without a single line of your server logic changing. That makes tool schemas exactly the kind of thing that belongs under test in CI, and exactly the kind of thing most teams don't test at all.

What actually counts as "the schema"

When a client calls tools/list, each tool in the response looks roughly like this:

{
  "name": "search_flights",
  "description": "Search available flights between two airports.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "origin": { "type": "string" },
      "destination": { "type": "string" },
      "date": { "type": "string", "format": "date" }
    },
    "required": ["origin", "destination"]
  }
}

inputSchema (and, if present, outputSchema) is standard JSON Schema. That's the exact surface an LLM client uses to decide what arguments to pass and what shape to expect back. It's also the exact surface that tends to drift silently: someone renames date to travelDate while refactoring, adds a genuinely required field without updating required, or narrows a string to an enum and forgets that older clients pass free text.

Three checks worth running

1. Snapshot the tools/list output

The cheapest test is a straight snapshot. Boot the server (or a test instance of it), call tools/list, and diff the result against a committed golden file.

def test_tool_schema_snapshot(mcp_client, golden_tools):
    result = mcp_client.call("tools/list")
    assert result["tools"] == golden_tools

This catches every change, including ones you meant to make. That's the point — it forces a human to look at the diff and consciously update the golden file, rather than a schema change shipping as a side effect of an unrelated refactor.

2. Diff schemas semantically, not just textually

A raw diff flags every change equally, which trains people to rubber-stamp the PR. It's worth separating changes into two buckets:

• Backward compatible: adding a new optional property, adding a value to an existing enum, loosening a constraint (e.g., removing a maxLength).
• Breaking: removing a property, adding to required, renaming any field, narrowing a type, or removing an enum value.

A small script using a JSON Schema diff library (or even hand-rolled comparison of the required arrays and property sets) can classify a change and fail CI only on the breaking bucket, while just logging the compatible ones for visibility.

3. Validate the schema is actually valid JSON Schema

This sounds redundant until it happens to you: a hand-written inputSchema with a typo'd keyword, a required entry that references a property that doesn't exist, or a type that isn't a valid JSON Schema type. None of these necessarily break at runtime — some clients are lenient — but they're bugs, and they're free to catch with a validator:

from jsonschema.validators import validator_for

def test_input_schemas_are_valid_json_schema(mcp_client):
    tools = mcp_client.call("tools/list")["tools"]
    for tool in tools:
        schema = tool["inputSchema"]
        validator_cls = validator_for(schema)
        validator_cls.check_schema(schema)

Wiring it into CI

None of this needs special infrastructure — it's a normal test job that happens to boot your server first:

name: mcp-schema-check
on: [pull_request]
jobs:
  schema-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run start:test-server &
      - run: npx wait-on tcp:3000
      - run: npm run test:mcp-schema

The only real discipline required is treating the golden snapshot file like an API contract: changes to it should be a deliberate, reviewed part of the PR, not an incidental diff nobody reads.

What CI can't catch

CI tests the version of the server you're about to ship. It says nothing about the version that's actually running in production right now, which might differ due to a rollback, a config flag, a canary deploy, or a dependency that resolved differently at build time than it did last week. That's a separate problem from testing — it's monitoring, and it needs to run continuously against the live endpoint rather than once per PR. MCP Vitals does that half: it performs the real initialize → tools/list handshake against a running server on a schedule and diffs the schema between checks, so drift that CI never saw — because it happened after deploy, not before — still gets caught and alerted on.

CI tests catch the drift you're about to introduce. Production monitoring catches the drift that slipped past everything else. A reliable MCP server benefits from both, not one instead of the other.