The leaderboard forgot the transmission.
HarnessTax is a useful correction to model-only shopping. Its dashboard pairs the same named model across Pi, Claude Code, and Codex harnesses on SWE-bench Lite and Terminal-Bench 2, then shows success and cost together. The important word is pairs: model, task cohort, and repetitions are held close enough to inspect the harness effect instead of comparing unrelated headline scores.[1]
One published SWE-bench pairing for Claude Fable 5 covers 30 tasks over three repetitions. Claude Code versus Pi showed no clear success-rate separation after the dashboard’s multiple-comparison treatment, while the reported mean cost ratio was about 2.0 with a 95% interval of 1.76–2.33. Codex versus Pi on the same pairing likewise showed no clear success-rate difference, with a reported cost ratio around 1.34.[2]
That is not a universal harness ranking. It is a better question: what did this assembly buy on this matched cohort, and what did it burn? The Terminal-Bench comparisons are explicitly labeled descriptive common-30 estimates without confirmatory tests. Keep that caveat attached; a dashboard is not permission to sand off uncertainty.
The harness can spend more, retry differently, expose better tools, or fail more gracefully—and every one of those choices can masquerade as “model quality.”
The IDE is becoming a harness socket.
VS Code 1.138 makes the assembly visible. Its Agent Host can run Codex with Copilot- or ChatGPT-backed models, continue a session across apps, attach a quick chat to a folder or isolated worktree, use built-in, extension, and MCP tools, and run local sessions in a Dev Container. That is not one feature. It is a matrix of model source, session host, tool surface, workspace attachment, and runtime placement.[3]
The upgrade is flexibility. The operating hazard is silent configuration drift. If a result changes after switching surface, subscription, workspace, or execution floor, the model name is the least interesting line in the incident report.
Release notes are dyno sheets.
Claude Code 2.1.274 is packed with harness-level behavior: a bounded first-turn MCP startup wait, self-healing for some corrupted transcripts that caused endless retries, longer per-server MCP timeouts, context compaction fixes, preserved agent flags after auto-update, and leaner code-review prompting for models without tuned settings. None of those changes alters the weights. All can alter latency, cost, completion, or failure shape.[4]
This is why “we tested Model X” is weak release evidence. Record the harness version, resolved configuration, connected tools, environment identity, turn and retry ceilings, and exact acceptance oracle. When the behavior moves, diff the chassis before blaming the cylinder.
Good harnesses manufacture disagreement.
Cloudflare’s open security-audit skill makes the control logic concrete. It separates reconnaissance, coverage-led hunting, candidate validation, structured findings, independent record verification, and reporting. The finder does not certify its own finding. Confirmed, needs-validation, and rejected are separate states. Validators run again after material replacements.[5]
That architecture matters more than the swagger of “multiple agents.” Parallel agreement can be correlated enthusiasm. A useful harness assigns different jobs, preserves rejected candidates, validates machine-readable records, and gives uncertainty somewhere honest to live.
Buy the lap, not the logo.
For an adoption decision, run the same tasks with the same model build, same environment, same permissions, same task order, and the same stopping rules. Price the whole rollout, not just successful calls. Capture success, wall time, tokens or spend, turns, tool errors, retries, changed files, and the final oracle. Repeat enough to see variance.
The swap stand below replaces a model dropdown with a comparison mode plus four parity locks. It executes no benchmark and invents no telemetry. Its readiness dial is a teaching proxy; closing every lock produces a review gate ready plan, not a winning harness.