Building a better MCP server and proving it

I'll pull the ClickHouse article so the paragraphs stay grounded in real names and mechanics, without inventing versions or figures.ClickHouse engineer Brandon Pereira published a deep dive on how the team builds and grades the ClickStack MCP server for SRE-style agents. At Open House in San Francisco they introduced purpose-built observability tools so agents investigate production incidents with high-level primitives instead of assembling raw SQL. Stage numbers against a generic ClickHouse SQL interface were concrete: 18% more accurate root-cause and remediation outcomes, 26% fewer tool calls, and 2.4x more consistent results across runs. The post explains the measurement system behind those claims: hdx-evals, an open-source framework in the HyperDX repository that seeds identical synthetic telemetry, runs Claude agents against each MCP and model pair, and scores answers blindly.

The architecture split is deliberate. The ClickHouse MCP exposes SQL access, so the model must discover schema, write queries, and stitch multi-step investigations itself, which often means extra tool calls, high-cardinality dumps, and run-to-run drift. ClickStack MCP keeps SQL under the hood but ships semantic tools for event patterns, time-window comparison, outlier detection, and log-to-trace navigation. hdx-evals provisions eval accounts, clones real OTel tables (including a two-stage materialized-view rollup path), and generates tens of millions of logs and spans from a seeded PRNG so every run sees byte-identical data. Five scenarios cover different skills: error-root-cause (8 failing traces in 12M+ spans and 12M logs across 25 services, with a CDN distractor at 10x volume), latency-spike (p99 regression for enterprise tenants in 20M spans across 5,000 tenants), noisy-signals (16M logs where safe-to-drop and must-keep patterns share service and severity), service-health-check (36M events with no incident), and segmented-regression (Simpson’s Paradox across enterprise tier and cache miss). Each run is a sandboxed Claude Code process with a 15–25 tool-call budget, no schema handout, and a deny-list that strips non-investigation tools. Grading mixes weighted regex checks, a blinded LLM judge (60% of the quality mix), and a tool-error penalty capped at 20%, combined as clamp((0.4 × programmatic + 0.6 × judge) − penalty, 0, 1).

For engineers shipping agent tooling, the lesson is that MCP quality is a product surface you can regress with a renamed parameter or a vague error string. Tool schemas need enough parameters to be useful without enough surface for hallucination. Response design matters at the edges: oversized results should push the agent toward a tighter query; opaque first-call errors often cause the agent to abandon that tool for the rest of the investigation. Latency compounds because a slow early call breaks hypothesis-forming and reduces reuse. Sandboxing is not optional either; early runs showed Claude hunting the filesystem for prior outputs and ground truth, so each eval lives in an ephemeral temp dir with no bash, write, edit, glob, or webfetch.

On results, ClickStack beat the ClickHouse SQL MCP on all five scenarios under Claude Opus 4.6 with ten runs per cell. Combined scores: error-root-cause 93% vs 73% (+20pp), noisy-signals 64% vs 45% (+19pp), latency-spike 60% vs 43% (+17pp), segmented-regression 75% vs 60% (+15pp), service-health-check 61% vs 54% (+7pp). That gap is the competitive claim against raw-SQL MCP baselines in the same stack, not a vague market narrative: semantic observability tools outperform free-form SQL for agent-led incident work when measured on planted anomalies, distractors, and consistency. The framework also multiplies across models, so a new model can be scored on the same scenarios before production use.

Practical takeaway: treat agent MCP changes like production path changes and gate them on reproducible evals. hdx-evals is already public at github.com/hyperdxio/hyperdx/tree/main/packages/hdx-eval. What to watch next is CI integration (faster setup, pre-seeded ClickHouse Cloud DBs, containers, GitHub Actions on MCP diffs), plus broader scenario coverage beyond traces and logs into dashboards, alerts, and metrics.