Home / Blog / GitHub Copilot benchmarks site adds harness comparisons
Tech News

GitHub Copilot benchmarks site adds harness comparisons

GitHub launched a public preview update to its Copilot benchmarks site, adding Terminal-Bench-Science 0.1 and paired harness comparisons against Claude Code.

By Dillip Chowdary • Nov 18, 2026 • Source: GitHub Changelog

GitHub Copilot benchmarks site adds harness comparisons
Image: GitHub Changelog

At a glance

Product
GitHub Copilot benchmarks site
Update
Harness comparisons view
New benchmark
Terminal-Bench-Science 0.1
Harnesses compared
Copilot CLI, Claude Code, Codex CLI
Reasoning level
Medium reasoning effort
Status
Public preview

GitHub launched a public preview update to its GitHub Copilot benchmarks site, introducing a dedicated harness comparison view and scientific research evaluation through Terminal-Bench-Science 0.1. According to GitHub Changelog's report, the refreshed benchmarking platform allows developers to compare how distinct agent runtimes execute tasks using identical underlying models. The tool tracks performance metrics across software engineering problems, weighing task resolution rates, reasoning effort levels, and mean financial cost per task.

This update directly serves engineering leads, tool builders, and developers evaluating agent harnesses such as GitHub Copilot CLI, Claude Code, and Codex CLI. By isolating harness scaffolding from base model capabilities, the site reveals how runtime orchestration influences accuracy, variance, and operational spend. Readers can inspect head-to-head metrics to determine whether scaffold optimizations deliver measurable cost reductions and task completion gains across standard software workloads and scientific domains.

01Public preview harness comparison view

GitHub expanded its Copilot benchmarking site to evaluate agent harnesses alongside standard foundation models. The interface evaluates GitHub Copilot CLI directly against Claude Code and Codex CLI under standardized conditions. Every comparison controls for reasoning effort, isolating how the surrounding client implementation alters practical output.

The platform visualizes results using a dual-axis scatter plot and a paired differences analysis. The scatter plot maps resolution rates against mean cost per task for each harness configuration. Shaded ellipses highlight run-to-run variation observed across published runs, while connecting lines link matched harness evaluations to make divergence immediately visible.

The paired differences view calculates GitHub Copilot CLI's resolution advantage in percentage points alongside relative cost savings expressed as a percentage. Both metrics display approximate 95% bootstrap intervals to indicate statistical significance across repeated evaluations.

02Terminal-Bench-Science research evaluation scope

Coverage on the platform now reaches beyond traditional software engineering repositories into complex scientific workflows. The site added Terminal-Bench-Science 0.1, an evaluation suite designed to test AI agents against multi-step research problems across technical disciplines.

The benchmark evaluates agent capabilities across life sciences, physical sciences, earth systems, mathematics, and engineering domains. Tasks are authored and reviewed by domain experts within each field to ensure challenges reflect authentic computational research rather than synthetic programming exercises.

Developers can inspect these domain-specific outcomes by selecting Terminal-Bench-Science 0.1 from the site benchmark picker. The interface reports whether agents sustain coherent terminal reasoning across specialized data processing pipelines and research environments.

03Controlled model matching and exclusion parameters

The comparison architecture enforces strict pairing criteria to avoid mismatched model distortions. GitHub limits side-by-side harness analysis to identical model configurations operating at medium reasoning effort.

Unmatched runs and proprietary configurations without exact peer counterparts are excluded entirely from the harness view. For example, GitHub Copilot Fusion and unique harness permutations are omitted rather than benchmarked against dissimilar base models.

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

Metric or conditionCopilot CLI paired viewAlternative harnesses
Reasoning effort settingMediumMedium
Model matching requirementExact pairs onlyExact pairs only
Evaluated toolsGitHub Copilot CLIClaude Code, Codex CLI
Statistical error bounds95% bootstrap intervals95% bootstrap intervals
Unmatched configurationsExcluded (e.g. Fusion)Excluded

Note: Copilot-only configurations such as GitHub Copilot Fusion are excluded from paired harness comparisons to ensure every data point reflects identical base models.

04Inspecting published run data online

Developers can explore the complete dataset directly on the GitHub Copilot benchmarks site without installing local benchmarking tools. The web dashboard provides interactive filters to toggle between standard programming suites and scientific evaluation sets.

Users can isolate specific harnesses or examine multi-run variance across the published evaluation matrix. The interface groups runs so engineers can examine how client harnesses alter resolution accuracy on identical benchmarks.

  1. Access dashboard Navigate to the public GitHub Copilot benchmarks portal in a web browser.
  2. Select benchmark Choose Terminal-Bench-Science 0.1 or an established software suite from the benchmark picker.
  3. Open harness tab Switch to the Harnesses view to inspect paired scatter plots and bootstrap confidence intervals.

05Accessing benchmarked agent runtimes

The harness comparisons focus on command-line developer agents that run in terminal environments. Teams looking to test these environments locally can access GitHub Copilot CLI, Claude Code, or Codex CLI through their respective client update and launch commands.

To update or install Claude Code CLI for local agent workflows:

Command
npm install -g @anthropic-ai/claude-code

To switch models interactively or specify model selection within the agent interface:

Command
/model claude-fable-5-1

To launch a terminal session pinned to a specific model version directly from the command prompt:

Command
claude --model fable

Frequently asked questions

What does the new Harnesses view on the GitHub Copilot benchmarks site compare?

It compares GitHub Copilot CLI against Claude Code and Codex CLI using identical base models at medium reasoning effort, measuring resolution rate, reasoning effort, and mean cost per task.

What is Terminal-Bench-Science 0.1?

It is an evaluation suite added to the site that tests AI agents on scientific research workflows across the life, physical, earth, mathematical, and engineering sciences.

Why are configurations like GitHub Copilot Fusion omitted from the paired harness view?

The view only displays exact model pairs at medium reasoning effort, excluding unmatched configurations or Copilot-only setups to prevent misleading comparisons against dissimilar models.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →