Build-A-Chip-Bench Joins Fireworks’ Specialized Intelligence Index
“The RTL looks right” is not enough to evaluate frontier models.
A chip design can pass functional verification, yet still miss timing, exceed its area budget, or encounter routing failures. That gap is why we built Build-A-Chip-Bench, which is joining Fireworks’ Specialized Intelligence Index (SII). It brings semiconductor design into a broader effort to evaluate open, closed, and specialized models against the demands of professional work.
For chip design, the ultimate question goes beyond whether a model can write plausible RTL: can AI carry design intent through implementation, adapt its planning and execution to physical feedback, and iterate until the design meets both functional and physical requirements?
Why join the Specialized Intelligence Index?
General benchmarks have helped track AI progress, but their bounded tasks and standardized scoring capture only part of what we ask models to do.
SII brings together practitioner-built benchmarks to evaluate models on domain-specific tasks. It gives teams a way to examine capabilities that broad, general-purpose evaluations may leave unresolved.
To enable AI-native chip design, those unresolved questions are consequential. A faster RTL generator helps, but engineers still need to know whether the design choices can pass synthesis, verification, and physical implementation.
And that is the motivation for ChipAgents to join Fireworks’ SII, building a more complete picture of what today’s models can do, with the ambition of making AI more useful in each of those fields. Build-A-Chip-Bench gives semiconductor design a place in a wider conversation about applied AI.
It also creates a basis for investigating specialization. Domain-specific evaluation can expose where improvements are needed; it does not assume in advance which model or approach will provide them.
From workload specification to GDS
Developed by ChipAgents with NVIDIA contributions, Build-A-Chip-Bench evaluates AI agents from workload specification through GDS layout.
Each task provides a fixed interface, open-source tools, and an open-source process design kit (PDK). Agents can query a separate verification service but cannot inspect its hidden tests or verification IP. An independent grader evaluates the final submission.
Scoring has two gates:
- Functional verification: the design must pass checks against the hidden verification IP.
- Physical pass: the design must complete physical implementation with routing complete, no hold-time violations, and clean design-rule checking (DRC) results.
Submissions that pass both gates receive a task-specific score reflecting power, performance, and area (PPA) against the task’s constraints. Failing either gate yields zero. The overall score averages across tasks.
This structure rewards agents for producing correct designs and improving physical implementation quality. Merely reaching the end of the tool flow does not guarantee a strong score.
Read the research article for the detailed methodology.
What 16.6% means in the initial evaluation
In the initial evaluation, all models used the OpenCode agent harness. The leading model achieved a 16.6% overall score, with a 65% functional pass rate and a 75% GDS pass rate.
Those figures measure different things. Functional pass and GDS pass are reported separately across all tasks; neither rate alone establishes that a submission passed both gates. The overall score combines those gate outcomes with implementation quality.
16.6% means that, under this evaluation’s scoring rules, the combination of correctness, physical implementation, and PPA performance still leave substantial room for improvement.
Our ambition: make full-flow progress measurable
We want to help build AI that can carry design intent through the full design flow and use physics-grounded evidence from each stage to make better decisions. For instance, an agent needs to recognize when a timing problem calls for an architectural change, preserve functional behavior while optimizing PPA, and spend its execution budget on experiments that can meaningfully improve the design…
Build-A-Chip-Bench gives us a way to test those capabilities together.
We want it to become a shared research instrument for chip designers, model developers, and AI engineers: whether improvements come from stronger models, specialized training, or better orchestration. Explore Build-A-Chip-Bench and Fireworks’ Specialized Intelligence Index.
Frequently Asked Questions
What does Build-A-Chip-Bench evaluate?
It evaluates AI agents across a digital chip design workflow, from workload specification to GDS layout. Scoring combines functional correctness, physical implementation checks, and task-specific PPA performance.
How is it different from existing chip design benchmarks?
Existing benchmarks such as CVDP and ChipBench have provided valuable reference points for evaluating hardware AI. Build-A-Chip-Bench extends that work in two ways.
First, it tests a longer workflow. Point-task evaluations focus on selected stages, such as RTL generation or verification. Build-A-Chip-Bench follows decisions through physical implementation, where their downstream consequences become measurable.
Second, it evaluates implementation quality after correctness and physical checks pass. That creates room to distinguish agents that produce a passing design from those that can continue improving it under competing constraints.
What does a task look like?
A task starts with a workload specification and a fixed interface. The agent develops the architecture and microarchitecture, writes RTL, verifies its behavior, and carries the design through synthesis, floorplanning, placement, clock tree synthesis, routing, and the benchmark’s final checks.
It then submits the GDS layout and supporting design artifacts for independent grading.
Reference designs help calibrate achievable task constraints. Submissions must pass the functional and physical gates to receive a nonzero score; their task-specific PPA performance then determines that score. Matching or exceeding a human reference design is not a universal definition of correctness.
Why are the tasks hard?
The difficulty comes from the dependencies within chip design. An architectural choice can look reasonable in simulation and later create a timing bottleneck, excessive area, or routing congestion.
Solving those problems requires an agent to connect physical feedback to earlier decisions, judge whether to change the RTL or adjust the implementation, and verify that its fix preserves working behavior.
Tool execution also takes time. An agent must plan around slow synthesis and place-and-route runs, leaving enough time to inspect results and act on them. Repeated trial and error can exhaust the budget before the design converges.
Does the benchmark prove specialized models are better?
The evaluation makes that question testable. It does not assume that specialization will outperform a general-purpose model. Any advantage must be demonstrated under comparable conditions.