Every HW-Bench eval is built the same way, from a map of the work to a score tied to engineer hours.

The same seven steps apply to every task we add, and each step below has its own figure.

Mapping the universe

We map hardware engineering work by discipline (electrical, mechanical, software/firmware, systems/integration) and by stage (design and simulate, build and test, operate and sustain). A cell is one discipline at one stage. Within a cell, a task is a unit of useful work: the smallest complete job an engineer gets handed, ending in a deliverable someone can use.

Split by: Discipline and stage
A cell: One discipline at one stage of the life cycle
A task: One deliverable an engineer hands off

Fig. 01 · Map

Hardware work, split by discipline and by stage

Every cell holds tasks. We take one task at a time and break it into the atomic steps it is made of. Illustrative.

A cell: One discipline against one stage of the life cycle
A task: Within a cell, a unit of useful work
An atomic step: The steps that a task is made of

DisciplineDesign & simulateBuild & testOperate & sustain

Electrical engineeringDesign a 2.45 GHz ISM band antennaCorrelate SPICE against bench measurementDiagnose EMI failures reported from the field

Mechanical engineeringBuild a thermal conductivity map for a PCBCorrelate a thermal model against chamber dataDiagnose a unit overheating in service

Software / firmwareBring up a new radar sensor unitGenerate a driver for a peripheral sensorPatch an FPGA image already in the field

Systems & integrationDefine the board, firmware and enclosure interfacesProduce an end-of-line test plan for a PCBTroubleshoot a PCB returned from the field

Design a 2.45 GHz ISM band antenna, as atomic steps

Decomposing the task

We then break each task into the atomic steps an engineer takes to finish it: who does each step, which tools they use, what files they start from, and what they hand to the next person. Few tasks run in a straight line, so we also record where the work loops back. Partner SMEs review the decomposition.

Broken into: Atomic steps
Per step: Tool, input files, deliverable
Reviewed by: Partner SMEs

Fig. 02 · Decomposition

One task, split into steps across the tools an engineer uses

Each step has its input files, its tool and what it hands on. When a check misses, the work goes back a step. Illustrative.

Building the eval

Every eval is built from real files, in real tools. The prompt reads the way an engineer would hand the task over, and input files stay in their native formats. A task tested on its own gets the golden answers from earlier steps; a full end-to-end run gets the model's own earlier outputs.

Workbench: HFSS, KiCad, simulators, CAD and EDA tools, driven by the model directly
Input files: Native formats
Built by: Engineers who do this work in industry
Reviewed by: Partner SMEs

Fig. 03 · Eval

An eval is a real job, in real tools

Native input files go in, and the model drives the same tools an engineer would. Illustrative.

Running the eval

Every model gets the same bench: frontier and open-weight models run side by side, with the same harness, tool integrations, prompt and input files, and nothing tuned per model. Running both kinds shows a team whether a model it fully controls is good enough. Every atomic step runs five times per model, because one good run is not a capability.

Reasoning effort: High
Limits: 300 tool calls per turn, 3 hours per task, or 45 minutes with no progress
Context full: The run summarises its history and continues. Counted.
Timeout or cap hit: Re-run, not counted

Fig. 04 · Run

Five runs of the same step, logged as they work

Every run is recorded event by event until it finishes or hits a limit. Illustrative.

Grading

An engineer does each task for real, in the real tools; their output is the golden answer. Anything with a right answer, like a simulated frequency response or layout dimensions, is computed from the model's output files, so the same output always scores the same. Judgment calls, like reasoning, trade-offs or a test plan, go to an LLM judge.

Rubric: Written with that engineer: which criteria matter most, what counts as good
Graded against: The golden answer and the rubric
Judge agreement: Refined until it agrees with partner SMEs, who hand-grade a sample, over 95% of the time
Recalibrated: Whenever a rubric or task changes

Fig. 05 · Grading

Graded against the engineer’s own answer

A computed check where the output has a right answer, an LLM judge where it takes judgment. Illustrative.

Score

Accuracy alone does not make a business case. For every run we record accuracy, cost and wall-clock time, and across runs we measure how much the results move. We also change one input at a time, or take a tool away, to see whether a model flags what is missing, works around it, or makes something up.

Per run: Accuracy, cost, time
Across runs: Spread
Stress test: One input changed or removed

Fig. 06 · Score

A score is a spread, not one number

Five runs per model per step. How much they move matters as much as where they land. Illustrative.

Tie the scores to work decisions

For each task, we record how long it takes an engineer today to do it and the level of experience it requires. That shows where a model is worth using: which steps it can take over, which it can speed up, and which still need an engineer.

Per task: Engineer hours today
And: Experience it needs
Decision: Take over, speed up, or keep

Fig. 07 · Work decisions

Every task sits on two axes: the experience it needs and the time it takes

A model earns its place by moving a task on this board. Each dot stands for one task. Illustrative.

Move it left. The same person does it, in less of their time.
Move it down. A less experienced person can now do it.
Move it off the board. Nobody has to do it at all.

A public subset

We will release a subset of HW-Bench publicly: the tasks, the schematics and the results. Anyone can judge the quality of the tasks, run the tests and reproduce our results. Our first full results go public next month.

Private evals: Founding partners AMD, Anduril and SpaceX run the same evaluations on their own data, in their own environment. Their designs never leave the partner; only summarized findings are shared with the broader community.