Technique names are not magic words.
Dan Luu’s September 2026 experiment ran a Rust Zstd implementation task under 26 prompt conditions—including TDD, fuzzing, property-based testing, mutation testing, formal methods, named libraries, and no extra instruction. Most condition-and-effort cells used 80 runs. Nothing “wildly” outperformed, and the no-instruction default landed above the equal-condition average. More important than the leaderboard: inspection found agents often wrapped ordinary weak tests in the requested technique, proved irrelevant properties, or generated random cases dominated by invalid inputs.[1]
That does not prove that property tests, fuzzers, formal methods, or TDD are weak. It shows that invoking their names is not equivalent to applying their leverage. A torque wrench used as a hammer does not invalidate torque.
A test framework is a machine. The defect model is the driver.
The harness changes what “done” looks like.
A separate public hangar experiment sent the same self-contained Three.js prompt through ten model-and-harness combinations. The table records duration, token traffic, tool calls, tool errors, whether the result opened in a browser, and whether screenshots were checked. Some runs did both visual steps; some did neither.[2]
That table is a useful process receipt, not a universal quality ranking: one prompt, heterogeneous models, and no common correctness score cannot crown a winner. But it exposes something dashboards usually hide. “Generated a file” and “opened the file and looked at it” are different operational states. The harness decides whether inspection is natural, optional, or absent.
Autonomous repair loops need a sharper brake.
VS Code 1.136 introduced Agent Merge in preview: an agent can address review feedback, failed checks, and merge conflicts, rerun workflows, and repeat until the pull request is ready to merge. The same release organizes delegated chats and pending approvals across sessions.[3] GitHub also announced that Copilot code review can approve pull requests when its comments are resolved; GitHub says approvals do not override branch protection or required human approvals.[4]
These are real workflow upgrades. They also make weak checks more dangerous, because the loop can efficiently optimize toward whatever the checks reward. If the oracle only asks “does the code compile and satisfy examples?”, the repair loop becomes a very fast theater company.
Build a test contract before a test pile.
- Name the likely defect: boundary error, state leak, ordering bug, numerical drift, concurrency race, malformed input, compatibility break.
- Choose an independent oracle: reference implementation, executable specification, metamorphic relation, invariant, differential peer, or human-visible artifact.
- Attack the shape: boundaries, degenerate cases, structured mutations, long sequences, interruption points, and combinations—not random noise for its own sake.
- Preserve the smallest wreck: seed, fixture, environment, command, observed output, expected output, and the minimized counterexample.
- Keep approval outside the same blind spot: the actor that writes the patch should not be the only actor defining success.
Tell the agent to use tools, sure. Then inspect whether it used their leverage. The goal is not a more ceremonial green check. The goal is a failure that had somewhere to hide and got dragged into fluorescent light anyway.