Determinism has inputs
Celld runs production event-handling code inside a simulator that chooses the next request, asynchronous task, storage result, clock advance, and fault. The project says the same code, settings, and seed produce the same sequence and intermediate states.[1]
That sentence names the contract. The seed does not identify the code or the settings. It also does not describe the invariant that failed. A bug report that saves only the seed leaves the next engineer to guess the rest.
The replay unit is the execution contract, not the lucky number.
Separate choosing from doing
Celld separates the code that selects the next event from the code that handles it. The simulator can delay a storage write, make it fail, advance simulated time, or run another task before a pending request completes. A checker tests invariants after each action.[1]
One generated order exposed an alarm race. A delayed cleanup removed a wake entry after a newer alarm had reused it. Both client operations had returned success, but no wake entry remained to reactivate the cell. The invariant caught the mismatch between the alarm stored in SQLite and the missing wake entry.[1]
The reusable move is architectural. Put scheduling, fault choice, and time behind controllable interfaces. Keep the production handler on the other side. Then save every choice needed to replay the failure.
One process can contain a bad week
FoundationDB's Flow runtime supports production execution and deterministic simulated execution. Its simulator runs an entire cluster in one single-threaded process. It models machines, drives, networks, communication delays, restarts, and datacenter failures.[2]
A single-threaded simulator does not claim production has one thread. It gives the test harness control over execution order. FoundationDB says that determinism allows exact repeatability and controlled experiments. Different tests and seeds still matter because one repeatable schedule covers one route through a much larger state space.[2]
Agent throughput raises the receipt bar
Cockroach Labs describes a coding-agent workflow with separate planning, implementation, review, and discharge roles. GitHub issue labels carry workflow state. Structured notes carry each stage's record. The process requires a reviewed plan before code and sends rejected work back through an explicit loop.[3]
That workflow is evidence about Cockroach Labs' own migration tooling. It does not prove that the same roles will work in another repository. It does point at a practical pressure. When patches arrive faster, failures need better records. A reviewer should be able to hand a simulation failure back with the exact revision, settings, seed, action budget, fault set, invariant, and observed step.
Ship the failure as a cassette
Use a replay packet with fields that can be checked rather than a prose note that says "flaky under seed 184467." At minimum, record:
- The source revision and simulator revision.
- The complete settings or a digest of an immutable settings file.
- The seed, random-number generator version, and action limit.
- The permitted requests, faults, delays, and clock behavior.
- The invariant name and the first step where it failed.
- The replay command and the result after the proposed fix.
Keep the first failing packet immutable. A fixed run is a second record. If the fix changes the settings, action budget, or invariant, say so instead of calling the new run a replay.