First Silicon
Can frontier models find real hardware bugs without raising false alarms? We collected 86 bugs that engineers found and fixed in open-source chip designs, hid which version each model sees, and let a formal checker decide.
| Opus 5.5max effort | Opus 5.5default | Fable 5.1 | Sonnet 5.5 | Haiku 5.5 | |
|---|---|---|---|---|---|
| Caught, zero false alarmsSTRICT · all 86 bugs | 18.6%16 / 86 | 14.0%12 / 86 | 9.3%8 / 86 | 8.1%7 / 86 | 3.5%3 / 86 |
| Without defective promptsSTRICT · 80 bugs | 17.5%14 / 80 | 15.0%12 / 80 | 10.0%8 / 80 | 8.8%7 / 80 | 3.8%3 / 80 |
| OpenTitanSTRICT · 70 bugs | 15.7%11 / 70 | 12.9%9 / 70 | 5.7%4 / 70 | 5.7%4 / 70 | 1.4%1 / 70 |
| Other open-source projectsSTRICT · 16 bugs in 10 projects | 31.3%5 / 16 | 18.8%3 / 16 | 25.0%4 / 16 | 18.8%3 / 16 | 12.5%2 / 16 |
| Caught, false alarms allowed86 bugs | 32.6%28 / 86 | 25.6%22 / 86 | 15.1%13 / 86 | 10.5%9 / 86 | 3.5%3 / 86 |
| Precision of alarmstrue alarms / all alarms | 34%49 / 145 | 32%34 / 105 | 22%21 / 94 | 32%15 / 47 | 21%6 / 28 |
| False alarm on the fixed designbugs · lower is better | 31.4%27 / 86 | 25.6%22 / 86 | 24.4%21 / 86 | 12.8%11 / 86 | 9.3%8 / 86 |
| Fixed after June 2026STRICT · 3 bugs, after training cutoff | 0 / 3 | 0 / 3 | 0 / 3 | 0 / 3 | 0 / 3 |
The best configuration, Claude Opus 5.5 at maximum effort, catches 16 of the 86 bugs cleanly, and two of its alarms in three are false. Most bugs escape every model: 67 of the 84 genuine fixes are caught cleanly by none of the four models at default settings.
With 86 bugs, the benchmark separates large models from small ones (Opus 5.5 against Haiku 5.5: 10 bugs against 1, p = 0.012 in an exact McNemar test), but not the strongest from each other (p ≥ 0.23).
Real bugs. 70 come from OpenTitan and 16 from 10 other projects. Each fix provably changes behaviour at the outputs, as certified by a formal miter; a trivial template (“outputs are zero after reset”) does not catch it; and none was used to train our own models. Two later turned out to be intentional behaviour changes, which we keep and flag.
A blind task. The model reads a documentation excerpt, the port and signal lists, and one version of the module, buggy or fixed, without being told which. It writes 3 to 8 SystemVerilog assertions describing the intended behaviour. File names are neutral, and the two versions of a bug never share a batch.
An exact judge. Yosys and SymbiYosys run every assertion on both versions, with bounded model checking and then a PDR proof. A failure on the buggy version is a catch only if the same assertion is proven on the fixed one; any failure on the fixed version is a false alarm. Under STRICT, a single false alarm cancels the catch.
The name comes from the first chips back from the fab, where a missed bug costs a new mask set and months of delay.
We let Opus 5.5 call the judge’s bounded model checker up to five times, on the shown version only. The method was designed on 20 bugs and confirmed once on 61 others.
The checker did not close the gap. When an assertion received a counterexample, the model dropped or changed it 79 % of the time, including 8 assertions that had correctly exposed a bug: it found the bug, the tool confirmed it, and the model decided its own assertion was wrong.
| 61 held-out bugs | STRICT | Precision |
|---|---|---|
| Opus 5.5 | 9 | 28 % |
| Opus 5.5, max effort | 12 | 32 % |
| Opus 5.5 with the checker | 10 | 22 % |
| Opus 5.5, checker and triage | 11 | 38 % |
The triage keeps a counterexample only when two independent reviewers quote, word for word, the sentence of the documentation it contradicts. It is more precise than random removal and stable across passes, but it did not meet our pre-registered criteria. The open problem is judging a counterexample: is the code wrong, or the assertion?
Squares, left to right: Opus 5.5 max, Opus 5.5, Fable 5.1, Sonnet 5.5, Haiku 5.5, with the checker, with checker and triage.
The designs are public, and some answers refer to remembered upstream code, which can only inflate scores; only 3 bugs were fixed after the models’ training cutoff, and none is caught. Six prompts had an enum table that did not match the source; without them, STRICT is 14, 12, 8, 7 and 3 out of 80. Each prompt was run once, the judge works at module level with a restricted assertion subset, and the benchmark has not yet been externally validated.
We publish the outcome of every bug, five bugs in full, and a salted SHA-256 commitment for each bug and prompt; the salt is revealed on audit. The other 81 bugs stay private, so the benchmark stays usable for future models. We can evaluate yours.
Want your model evaluated on First Silicon, or your own design checked? Write to us below, or at contact@umwelt.vision.