The compiler only checks the doorway.
EEBench’s power-loss task is small enough to understand and nasty enough to matter. A residential meter loses its 5 V input. Its processor needs 20 ms to save the accumulated reading, and the protected rail must remain above a 3.0 V brownout floor. One submitted design chose a nominal 22 µF ceramic capacitor. The source built. The circuit did not.
At 4.7 V bias, EEBench reports that part providing only 11.4 µF of effective capacitance—far below the task’s 545 µF requirement. In transient simulation, the rail crossed below 3 V after 0.85 ms. The green build proved that names, types, and connections were legal. Physics supplied the rejection letter.
Generated engineering needs an executable world, not a confident paragraph.
Text wins when it carries constraints.
EEBench uses atopile rather than asking the agent to spend most of its context clicking through CAD chrome. Atopile describes itself as a language, compiler, and toolchain for electronics: declarative modules, interfaces, units, tolerances, and assertions that can update a KiCad layout. That shifts the agent’s attention from cursor coordinates toward components, connections, budgets, and failures.
The useful trick is not “hardware as code” by itself. Text can be wrong with tremendous efficiency. The win is that the same representation can be built, simulated, checked, diffed, and revised. A failed waveform becomes machine-readable steering instead of a red-faced lab surprise.
Make the bad corner the main lane.
EEBench says every specification becomes a deterministic check, using circuit graphs, BOM constraints, SPICE measurements, and worst-case component-tolerance corners. Its public scoring weights technical performance at 65% and cost efficiency at 35%, with cost credit only after the design works. That ordering is healthy: cheap smoke is still smoke.
The benchmark’s own boundary matters. Version 1 concentrates on requirements, design, and test in simulation. It does not yet establish that an agent can lay out, fabricate, bring up, and validate a complete physical product. The project says those harder layers are planned. Until then, read the score as evidence of a closed simulated loop—not a robot electrical engineer in a box.
The garage recipe.
- Write the requirement as a measurable limit. “Should survive” is vibes. “Rail stays above 3.0 V for 20 ms” is a gate.
- Separate nominal from effective. Model bias, tolerance, temperature, package, and operating point before trusting the label.
- Exercise the transient. Build success and steady state do not prove startup, dropout, recovery, or fault behavior.
- Price only working candidates. Economic optimization belongs behind the technical interlock.
- Keep the artifact trail. Requirement, source, part model, waveform, failing limit, revision, and final measurement travel together.
The Hacker News thread adds useful shop-floor texture but not benchmark-grade evidence. Experienced commenters describe promising boards, repairable errors, and the need for manual footprint and layout review. Treat those as field anecdotes. The stronger signal is the shape of the workflow: people are not merely chatting with a model; they are wrapping it in tools that can say no with measurements.