The Current Open RTL Benchmarks Are Finished

Authors
Surya Gunukula†, Jerry Yicheng Wang†, Kexun Zhang
† Equal contribution
Published
November 2026

Abstract

We audit 653 tasks over five open-source RTL benchmarks, CVDP non-agentic, agentic and heavy, RTLLM v2.1 and ChipBench, by translating each task's spec, reference RTL and hidden tests into Lean 4 and checking them against each other. Only 15% of tasks have a spec and tests that agree. For the rest, we use a QA agent to fix the spec, the tests or both, and have experts review every fix. Fixing these errors raises pass rates by 14 to 23 points across all four models we test: Claude Opus 5.5 goes from 77% to 96%, and Kimi K3.1 Pro from 69% to 92%. Today's open RTL benchmarks are therefore close to saturated, with little room left to measure further progress. The audit also exposes contamination: Opus passes 75 of the 160 tasks whose spec leaves out something the tests require, often by recalling the open-source design or practice problem the task was built from. We release the verified benchmarks on Hugging Face, and argue that RTL benchmarks need formal audits, private held-out subsets to limit contamination, and broader tasks.

1. Motivation

Benchmarks are the cornerstone of progress in AI, but a score only means something if a failure reflects a gap in the model's skill. Software engineering ran into this with SWE-bench (Jimenez et al., 2024). When OpenAI had developers review its tasks for SWE-bench Verified (OpenAI, 2024), 38.3% of samples had underspecified problem statements and 61.1% had tests that could reject valid solutions. RTL benchmarks are now used the same way. RTLLM (Lu et al., 2024) has been cited more than 170 times, CVDP (Pinckney et al., 2025) nearly 80 times in its first year, and CVDP is the basis of Si2's public LBC-bench leaderboard. None of them has had a comparable review.

A benchmark can be judged on three axes:

  1. Validity: it measures the skill it claims to measure.
  2. Fairness: the spec states everything the tests check.
  3. Verifier quality: the tests pass correct designs and fail incorrect ones.

Validity is hard to judge from the outside, because it depends on what the benchmark's authors meant to measure. Fairness and verifier quality depend only on the spec and the tests, so they can be formally checked.

2. Methodology

We translate the spec, the RTL and the tests into formal models in Lean 4. An agent writes the proofs relating them, and the Lean kernel checks them. Every conflict Lean finds comes with a proof.

When the spec and the tests conflict, a QA agent decides whether to fix the spec, the tests, or both (Section 3). To maintain quality, we ask human experts to review every fix by hand.

01
Inputs
SpecNatural-language task
RTLReference design
TestbenchHidden tests
02
Formalize
Spec modelLean 4
RTL modelLean 4
Test propertiesLean 4
03
Prove
Proof agentWrites the Lean proofs relating the three models
04
Check
Lean 4 kernelChecks every proof
no conflictconflict
05
Resolve
Task keptSpec and tests agree
QA agentFixes the spec, the tests, or both
Expert reviewEvery fix reviewed by hand
Figure 1. The spec, RTL and testbench are each translated into formal models in Lean. An agent writes the proofs relating them, and the Lean kernel checks them. Tasks with no conflict are kept as they are; conflicts go to a QA agent, and experts review every fix.

3. Outcomes of the formal check

Once Lean has checked the formal model of the spec against the formal model of the tests, each task falls into one of four cases, which decides the fix.

3.1Four cases

What the spec asks forWhat the tests check
Case 1MatchDo nothing
Case 2Weak testsHarden the tests ↓
Case 3Unfair testsUpdate the spec ↑
Case 4Both incompleteModify both ↕
Figure 2. The four outcomes of the formal check. The solid orange circle is what the spec asks for and the dashed blue circle is what the tests check. How the two overlap decides the fix: none, harden the tests, update the spec, or modify both.

3.2Case mix by benchmark

Before any model is rerun, a benchmark's mix of cases predicts which way its pass rates should move. Hardening weak tests (case 2) removes passes that only the weak tests allowed, so a benchmark heavy in case 2 should see pass rates go down. Completing the spec (case 3) removes failures caused by facts the spec never stated, so a benchmark heavy in case 3 should see them go up.

Case 1 · MatchCase 2 · Weak testsCase 3 · Unfair testsCase 4 · Both incomplete
CVDP non-agentic
4%
40%
22%
34%
CVDP agentic
3%
33%
22%
42%
CVDP heavy
7%
33%
29%
31%
RTLLM v2.1
24%
48%
18%
10%
ChipBench
51%
10%
32%
7%
Figure 3. Share of tasks in each case, per benchmark.*
* For CVDP agentic and non-agentic, we use the non-commercial subsets: the tasks that run on open-source tools rather than licensed commercial EDA tools.

3.3Examples

cvdp_copilot_signal_correlator_0015

Case 2 · Weak tests · CVDP non-agentic

The spec defines a weighted sum, +2 per matching bit and clamped at 15, with a worked example. The only test drives all ones and expects 15, so a design that always outputs 15 passes.

Testbench
Before
dut.input_signal.value = 0b11111111  # All bits `1`dut.reference_signal.value = 0b11111111  # All bits `1`await RisingEdge(dut.clk)  # Wait for one clock cycle…output_value = int(dut.correlation_output.value)assert output_value == 15, (    f"Clamping test failed: Expected 15, got {output_value}")
After
def expected_correlation(a, b):    return min(2 * bin(a & b & 0xFF).count("1"), 15)…vectors = [(0b10101010, 0b10101010),   # card example -> 8           (0x00, 0x00), (0xFF, 0x00), (0x0F, 0xF0),           (0x01, 0x01), (0x7F, 0x7F), (0xFE, 0x7F), (0xF0, 0xFF)]vectors += [(random.randint(0, 255), random.randint(0, 255)) for _ in range(40)]for i, (a, b) in enumerate(vectors):    await apply_and_check(dut, a, b, f"Vector {i}")

cvdp_copilot_digital_dice_roller_0001

Case 3 · Unfair tests · CVDP non-agentic

The tests expect dice_value to be 1 in reset, drive a port named reset, and read a parameter DICE_MAX. The spec says 000, names the port reset_n, and has no parameters.

Spec
Before
- `reset_n`: Asynchronous active LOW reset. Keeps the `dice_value` at 000 when LOW.…- `dice_value[2:0]`(3-bit): A 3-bit output representing the dice result, with valid values from 1 to 6.
After
**Parameter:**- `DICE_MAX`: default 6. The highest dice value.…- `reset`: Asynchronous, active-low reset. While it is low, `dice_value` is 1. After reset is released, `dice_value` stays 1 until the first roll ends.…- `dice_value[2:0]`: The dice result. It is always in the range 1 to `DICE_MAX`, and it holds its value while `button` is low.

adder_pipe_64bit

Case 4 · Both incomplete · RTLLM v2.1

The tests instantiate the adder with two parameters the spec never names, so a spec-faithful module fails to elaborate. In the other direction, the spec describes a pipelined adder, but the tests never send new operands on consecutive cycles; the hardened tests add 64 cycles of back-to-back operands.

Spec
Before
    adder_pipe_64bitInput ports:
After
    adder_pipe_64bitParameters (declared on the module, overridable at instantiation):    DATA_WIDTH = 64: operand width    STG_WIDTH = 16: number of operand bits added in each pipeline stageInput ports:

4. Results

The verified benchmarks are close to saturated.

Published spec and testsVerified
CVDP non-agentic
0
50
100
91
77
90
62
Claude Opus 5.5Claude Haiku 5.5Kimi K3.1 ProNemotron 3.5 Super VL
CVDP agentic
0
50
100
98
86
92
63
Claude Opus 5.5Claude Haiku 5.5Kimi K3.1 ProNemotron 3.5 Super VL
CVDP heavy
0
50
100
99
92
97
68
Claude Opus 5.5Claude Haiku 5.5Kimi K3.1 ProNemotron 3.5 Super VL
RTLLM v2.1
0
50
100
100
98
98
92
Claude Opus 5.5Claude Haiku 5.5Kimi K3.1 ProNemotron 3.5 Super VL
ChipBench
0
50
100
100
93
90
73
Claude Opus 5.5Claude Haiku 5.5Kimi K3.1 ProNemotron 3.5 Super VL
Figure 4. Pass rate per model and benchmark (a task passes at 2 of 3 trials), published vs verified. The number above each orange bar is the verified pass rate; hover a bar for exact counts. Claude models ran in Claude Code; Kimi and Nemotron ran in OpenCode via Fireworks.
ModelPublishedVerifiedGainPass rate
Claude Opus 5.5501624+12376.7% → 95.6%
Kimi K3.1 Pro447598+15168.6% → 91.7%
Claude Haiku 5.5411554+14362.9% → 84.8%
Nemotron 3.5 Super VL351441+9053.8% → 67.5%

Table 1. Tasks passing across all five benchmarks.

Every model gains between 90 and 151 tasks, and the ranking does not change. Kimi gains the most, halving its gap to Opus from 54 tasks to 26. The largest published gap is on ChipBench, where Opus passes 119 tasks against 88 for Kimi and 87 for Haiku. After hardening that gap shrinks to 10–13 tasks, in line with the contamination in section 05.

Nemotron 3.5 Super VL trails mainly because of agentic ability rather than hardware knowledge: it often fails to use the harness correctly.

5. Contamination in open-source benchmarks

Open-source benchmarks, especially those built from open-source designs, are heavily contaminated. Case 3 makes this visible: the spec leaves out something the hidden tests require, and the model never sees those tests, so it has no way to know the missing detail. Claude Opus 5.5 passed 75 of the 160 case-3 tasks anyway, and in many of them its reasoning names the open-source design or benchmark the task came from. Where we swapped in a different, equally reasonable value for the missing detail, the passing designs failed.

5.1Tasks built from open-source designs

Several CVDP agentic and heavy tasks are adapted from public GitHub projects. On these, Opus recognises the project and rebuilds the original module, including details the spec never states. In verilog_pcie_0004, taken from Alex Forencich's verilog-pcie, it reproduced a one-cycle output lag that contradicts the spec text, and every passing design fails a second test without the lag. In queue_0001, taken from RoaLogic's rl_queue, it used the original's default queue depth, and two of three runs tried to download the original. In sigma_delta_audio_0001, taken from the Minimig paula_audio_sigmadelta module, it rebuilt the whole datapath from memory, including a bug in the original.

  • “This is Alex Forencich's verilog-pcie library.”
    Claude Opus 5.5 on verilog_pcie_0004
  • “The spec closely resembles a known open-source fall-through queue design (RoaLogic rl_queue).”
    Claude Opus 5.5 on queue_0001
  • “Its names match the Minimig paula_audio_sigmadelta module one-for-one (ldatasum → load_data_sum, int_cnt → integer_count …). So I implemented that algorithm.”
    Claude Opus 5.5 on sigma_delta_audio_0001

5.2Tasks built from public exercises

Many ChipBench tasks come from Nowcoder practice problems, and some reuse RTLLM reference designs almost directly. In 24 tasks, Opus or Kimi named the source in its reasoning. Opus then followed the original answer even where it contradicts the spec: on gen_sc_traffic_lights, it overrode the spec's reset behaviour to match the RTLLM reference, and a test written from the spec fails all three of its passing designs.

  • “The skeleton's port list is copied exactly from the RTLLM traffic_light benchmark, and the clarification describes that benchmark's timing.”
    Claude Opus 5.5 on gen_sc_traffic_lights
  • “The hidden testbench will grade the design cycle by cycle, so I'm matching the well-known Nowcoder signal-generator structure, including how it handles a mid-waveform change of wave_choice.”
    Claude Opus 5.5 on gen_sc_signal_generator

Kimi does the same. Its reasoning names Nowcoder on various ChipBench tasks and reproduces the original Chinese problem text from memory, down to timing rules the spec never states. Haiku 5.5 and Nemotron 3.5 Super VL never name Nowcoder on any benchmark.

  • “I have seen HDLBits-like questions on Nowcoder: '数据位宽转换' description: 实现数据位宽转换电路,实现24bit数据输入转换为128bit数据输出。其中,先到的数据应置于输出的高bit位…”
    Kimi K3.1 Pro on dbg0_timing_non_integer_data_width_conversion_24to128
    English translationI have seen HDLBits-like questions on Nowcoder: ‘Data width conversion’ description: Implement a data width conversion circuit that converts 24-bit input data into 128-bit output data. Data that arrives first should be placed in the high bits of the output…
  • “Could be training code from '牛客网 Verilog 序列检测' … Common problem '状态机-1011序列检测' … 'flag信号在检测到序列的下一个时钟周期拉高,并持续一个时钟周期'”
    Kimi K3.1 Pro on dbg0_timing_state_machine_overlapping_sequence_detection
    English translationCould be training code from ‘Nowcoder Verilog sequence detection’ … Common problem ‘State machine: 1011 sequence detection’ … ‘The flag signal goes high in the clock cycle after the sequence is detected, and stays high for one clock cycle’
ChipBench problemRTLLM problemSimilarityChipBench tasks
traffic_lightstraffic_light0.905
asynchronous_fifoasyn_fifo0.814
simple_implementation_ramRAM0.784
johnson_counterJC_counter0.714
pipeline_multipliermulti_pipe_4bit0.684
width conversion 8 to 16width_8to160.634

Table 2. Token similarity between ChipBench reference designs and RTLLM reference designs, with comments and module names removed. Each shared problem appears as one generation task and up to four debug variants.

5.3Memorizing the benchmark itself

RTLLM and its reference designs have been public on GitHub since 2023. On 40 of its 50 tasks, Opus 5.5's reasoning names RTLLM, and it often recalls what the hidden testbench checks. Haiku 5.5 does so on 4 tasks and Kimi K3.1 Pro on none. Opus also knows where RTLLM's problems came from: it recognises some as simplified Nowcoder problems, down to the problem number.

  • “Now I'm recalling the RTLLM testbench, which I think only checks addresses 0 through 3 against the preloaded values.”
    Claude Opus 5.5 on ROM
  • “I'm recalling the original Nowcoder version of this valid-ready accumulator, which is a different problem from the simplified RTLLM one, so I'll set the handshake details aside for now.”
    Claude Opus 5.5 on accu
  • “I'm recalling the standard reference solution for the asynchronous FIFO problem on 牛客 VL45, which pairs the FIFO with a dual-port RAM submodule.”
    Claude Opus 5.5 on asyn_fifo
    English translationI'm recalling the standard reference solution for the asynchronous FIFO problem on Nowcoder VL45, which pairs the FIFO with a dual-port RAM submodule.

6. Future thoughts

After verification, frontier models are close to the ceiling on these benchmarks: across all 653 tasks, Claude Opus 5.5 passes 95.6% and Kimi K3.1 Pro 91.7%. Yet the benchmarks measure only a slice of RTL design. Measuring the rest means revisiting each of the three axes from Section 1.

6.1Validity: which skill are we measuring?

Today's tasks measure spec-following. ChipBench's cpu_div expects one specific divider, OpenMIPS's, quirks included. A design task would state the function and a throughput target and let the agent choose among correct architectures, each trading latency, throughput and area (Table 3). That choice is the skill these tests never see.

DividerLatencyNew divide everyArea
Radix-2 iterative~32 cycles~32 cyclesSmallest
Radix-4 iterative~16 cycles~16 cyclesLarger
Pipelined array~32 cycles1 cycleLargest

Table 3. Three correct 32-bit dividers.

6.2Fairness: how much should the spec state?

Some behaviour a spec leaves open is a design decision, not a spec bug. CVDP's queue_0001 (Section 5.1) never states its default depth, and its tests fail any depth of 5 or more, though depth is a plain trade of area for performance (Table 4).

Depth 2Depth 8
Storage2 entries8 entries, 4× the area
Absorbs a burst of2 items, then stalls the sender8 items
Hidden testPassFail

Table 4. Two default depths for queue_0001.

In a real project, the designer sizes the queue for the expected traffic and writes that assumption into the spec or datasheet. A fixed evaluation has no such channel, so the tests silently pick one answer. Letting the agent declare its assumptions would help, if someone can judge which are reasonable.

6.3Verifier quality: which designs count as correct?

Today's problem is weak tests: 214 of the 653 audited tasks had tests that could not fail a wrong design (case 2). Once timing is left open, the tests must also pass every valid design, and the two goals conflict (Figure 5). A cycle-exact check rejects a valid latency; a check on arrivals only passes a design that never responds. The target is a bounded check on every transaction.

Cycle012345Cycle-exactBoundedArrivals only
Requestreq
Design ArespPassPassPass
Design BrespFailPassPass
Design CFailFailPass
exactToo strictTargetToo loose
bounded window
Figure 5. One request, three designs, three checks.

Such a suite has to be validated against a set of designs: correct designs that span the open choices must all pass, and mutants of each must all fail. Even then, pass/fail cannot say whether B's extra latency was worth it; that needs PPA.

6.4How much does the task pin down?

A task pins down what its spec requires and what its tests check; in a fair task the two match. The more it pins down, the easier fairness and verification get, and the less design skill it measures (Table 5).

TaskSystem boundaryInterface, protocolMicroarchitectureCycle timing
Today's benchmarksFixedFixedConstrained by cycle timingFixed
Function-level testsFixedFixedOpenOpen
Transaction-level tests, agent picks the protocolFixedTransactions fixed, protocol openOpenOpen
End-to-end tasks, e.g. Build-A-Chip-BenchFixedOpenOpenOpen

Table 5. What each kind of task pins down.

Every row fixes the system boundary, since the tests have to connect to the design. The open question is how far out that boundary can move while the tests stay fair and rigorous.

7. Takeaways

  1. 01

    Every benchmark should be formally audited

    Every benchmark we audited had tasks where the spec and the tests disagree, and fixing them moves scores by double digits: on CVDP non-agentic, Claude Opus 5.5 goes from 70% to 91%. Without an audit, part of every leaderboard measures the benchmark rather than the model.

    If you would like an open-source benchmark, or one you are about to release, formally audited, reach out to research@chipagents.ai.

  2. 02

    Keep held-out subsets private, or refresh them

    Models pass tasks they could only solve by recalling the open-source designs behind them (Section 5). Once a benchmark is public, its scores mix skill with memory. Tracking real progress needs benchmarks that are refreshed regularly, or held-out sets that stay private.

  3. 03

    Build broader benchmarks

    For frontier models, today's open RTL benchmarks are close to saturated (Section 4). Section 6 sketches what broader tasks need, up to end-to-end design tasks like Build-A-Chip-Bench.

8. Verified sets

We are releasing each verified benchmark with its fixed specs and tests, and will keep adding newly audited benchmarks.

Of the 392 tasks with weak tests, we added checks to 182. In the other 210, mostly in CVDP, we fixed the spec or the broken tests where needed but did not add checks for the spec behaviour the tests leave unchecked. These tasks are still passable from the spec, but their tests can miss a design that skips that behaviour. We plan to cover them in a later update.

References

  1. [1]Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O. and Narasimhan, K. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. ICLR 2024. arXiv:2310.06770
  2. [2]OpenAI. Introducing SWE-bench Verified. 2024. openai.com
  3. [3]Pinckney, N. et al.. Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification. 2025. arXiv:2506.14074
  4. [4]Lu, Y., Liu, S., Zhang, Q. and Xie, Z. RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model. ASP-DAC 2024. arXiv:2308.05345
  5. [5]ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design. 2026. arXiv:2601.21448
  6. [6]Gunukula, S., Wang, J. Y. and Zhang, K. Introducing Build-A-Chip-Bench: from workload specification to GDS. ChipAgents Research Blog, 2026. chipagents.ai
  7. [7]Si2. Introducing the LBC-bench Leaderboard: A New Public Benchmark for LLMs in Chip Design. 2026. si2.org