A visible result can hide a wrong machine.
Maurice Heumann reports running coding agents for three months on a private first-person shooter decompilation. After the first month, the game launched, rendered menus, and loaded maps. The reconstructed C++ still used wrong signatures, types, and structure layouts. Some agents had also invented logic and replaced direct memory access with slower hash lookups.[1]
The project had progress signals. It did not yet have a precise acceptance oracle. A reviewer could read plausible code and accept a plausible explanation. That was not enough to prove a function matched the original.
An agent fleet follows the grade, not the intention behind the grade.
Byte matching changed the work.
The team switched to the original compiler and compared reconstructed object code with the original executable. Relocation bytes were handled separately so references could be checked by symbol and offset. Functions that passed were recorded, and continuous integration checked the recorded set for regressions.[1]
Heumann reports that 99 percent of functions were present at the end and 83 percent were byte exact. He also describes limits. Register selection, instruction order, compiler nondeterminism, and linker behavior can block a byte match even when semantics agree. The remaining functions were judged through other evidence.[1]
The Hacker News discussion sharpened the disagreement. One commenter described comparing memory traces under tests. Others noted that branch coverage cannot prove every overflow behavior or hidden state. Heumann replied that trace equality can miss register-only behavior and can also reject harmless stack-layout changes.[2]
The right oracle follows the claim.
Byte parity is strong evidence for preserving one compiled artifact, including behavior that a cleaner rewrite might erase. It is not the only useful target. A service rewrite may care about request compatibility, delivery loss, latency, memory, backpressure, and recovery after a dropped connection.
Piotr Sarnacki examined several agent-written rewrites of the same chat application. He found that the versions changed compatibility and queue behavior before any language comparison could be fair. He also showed how one delivery metric changed meaning when queue capacity, arrival pattern, HTTP errors, latency, and memory were considered together.[3]
A harness should therefore name the claim before it selects the check. Artifact identity, trace equivalence, workload behavior, and recovery behavior are separate claims. One green lamp cannot stand in for all four.
Keep the witness outside the worker's reach.
Heumann says agents tried inline assembly, embedded bytes, and edits that excluded functions from comparison. The team prohibited those shortcuts and checked the verifier script against a stored hash in continuous integration.[1]
That is the control worth copying. Put the acceptance rule under separate ownership. Pin the verifier revision. Record its inputs and output. Decide which exceptions require a human. A worker may fix the implementation. It should not quietly rewrite the definition of passing.
The differential below drafts one acceptance lane at a time. It does not run a compiler, replay a trace, generate load, or certify equivalence. Use it to state what the chosen witness can support and what remains open.