The methodology behind HW-Bench

We map hardware engineering work along two axes: discipline (electrical, mechanical, software & firmware, and systems) and lifecycle stage (design & simulate, build & test, and operate & sustain). Each cell in the grid includes multiple jobs, the smallest complete units of work an engineer hands off as finished deliverables.

The universe of hardware work is nearly infinite, so we select a set of jobs that covers as much breadth as possible: different disciplines, lifecycle stages, tools, and difficulty levels, from single components to full systems. Engineers at our partner companies help prioritize jobs that are common across the industry and valuable for the wider hardware community.

In collaboration with subject matter experts at our partner companies, our hardware engineers decompose each job into tasks, representing the steps they would take to complete the work themselves.

To avoid test-set leakage (where future models train on public benchmark data and inflate their scores), HW-Bench keeps most of its data private and draws on two sources:

• Arena Physica–generated data: For each task, our engineers create the input data. These tasks are then reviewed by subject matter experts at our partner companies. For transparency, we release a subset of this data publicly, including prompts, input files, rubrics, and rollouts. The remaining data stays private and contributes to the results we publish.
• Partner-contributed data: Our partners provide fabricated designs, test data, and field performance results from their hardware systems. For security, evaluations on this data run in each partner’s own environment. The data remains private to the partner, but the results are included in what we publish.

Every task is written by an engineer with years of industry experience in the domain, and has five parts:

• Prompt: the instructions an engineer would give an agent to complete the task
• Input files: the native-format files an engineer would need for the task, such as HFSS projects, KiCad boards, and measured S-parameter files
• Tools: the simulators, CAD tools, EDA tools, and bench equipment an engineer would use, such as Ansys HFSS, Siemens NX, KiCad, and Python
• Golden answer: the solution the engineer produces with the same input files and tools given to the model, used as the reference for grading and sometimes as an input to the next task
• Rubric: the accuracy criteria that score the model’s output, and the diagnostic criteria that assess the model’s reasoning and tool use

Prompts are written the way an engineer would assign the work in industry, typically 200 words. Beyond specifying where to write the output, the prompts do not include hints, step-by-step instructions, or scaffolding. They also don’t expose the rubric or impose an output schema on free-form deliverables. The agent has to work out the approach, choose its tools, and decide what information it needs from the same task description an engineer would get.

Verifiability is the foundation of our grading methodology. Because hardware is governed by physics, most deliverables can be reduced to measurable quantities with defined acceptance criteria, such as resonant frequency, return loss across a band, trace width, or thermal margin.

When writing a rubric’s accuracy criteria, the engineer labels each criterion as verifiable or judgment-based. A verifiable check compares a measurable value from the model’s output against a fixed target, such as a spec limit or a design rule. Like a unit test in software, it runs in code and always returns the same pass or fail result for the same output. Checks run on the files the model actually produces, not the results it claims in its response. For example, if a model reports that its antenna meets spec, the check parses the configuration of the simulation the model ran and the return loss from its output, and evaluates both against the spec requested in the task. Most checks run in seconds, but some take hours, such as those that require hardware-in-the-loop testing.

Some criteria can’t be verified in code because the output has no single correct form, such as whether a test plan prioritizes the failure modes most likely to occur in the field. These are graded with a judgment-based check: an ensemble of LLM judges scores each criterion independently. For our initial results, the judges are Claude Opus 5.5, GPT-6 Sol, and Grok 4.7, with reasoning effort set to high. Each judge scores the criterion from 0 to 1, and the criterion passes if the three scores average at or above the set threshold.

We calibrate that threshold for each judgment-based criterion against our engineers’ grades. For each task, multiple engineers independently grade a sample of model outputs, without seeing the judges’ scores. Wherever the engineers’ grades disagree with the judges’, the threshold for that criterion is adjusted. Calibration finishes once the judges and engineers agree at least 90% of the time.

Accuracy is weighted by importance. Our engineers and our partners’ engineers tag each accuracy criterion as critical (4–5 points), important (2–3 points), or minor (1 point), so higher-weighted criteria have a larger impact on a run’s overall accuracy score.

Every run is graded on accuracy and measured on cost and latency:

• Accuracy: the only graded metric, measured as the weighted share of accuracy criteria passed
• Cost: total model API spend per run
• Latency: wall-clock time per run, including tool and simulation time

We plot accuracy against cost and latency to show which models deliver the most accuracy for their price and speed. This helps teams decide when a more expensive model is worth using, and when a cheaper or faster model is sufficient for their use case.

But accuracy, cost, and latency alone don’t determine whether a model can be trusted with real work, so we also analyze every result along three additional dimensions.

1. Reliability

A model that scores well on mean accuracy but swings widely between runs can’t be relied on in a workflow, since an engineer has no way to know which run they’re getting. To test for reliability, we run every task five times on each model, using identical inputs. We hold every model to the same five-run budget and do not add runs after seeing results, because the number of attempts itself shifts the scores: on SWE-bench Lite, repeated sampling raised one model’s solve rate from 15.9% with a single attempt to 56% with 250 (Brown et al., 2024). Instead, we report mean accuracy across the five runs, the spread between the best and worst run, and pass^k: the probability that k random runs of a task all pass every critical accuracy criterion.

2. Trustworthiness

A hallucinated value that looks right is one of the most dangerous errors in hardware, because it can pass design review and only surface after manufacturing. Hardware is ‘compile once,’ so catching a hallucinated value after manufacturing means another fab cycle. To test for trustworthiness, we run ablations in which we move, contradict, or remove one input value a task depends on, and then classify how the model responds. From most to least transparent:

• Alerts and waits for input: it catches and states the issue, withholding any answer that depends on the missing value
• Proceeds with disclosure: it works around the issue or assumes a value, flagging what it did
• Proceeds silently: it works around the issue or fabricates a value, without flagging it

3. Reasoning

An accuracy score shows whether a run succeeded, but not why, and a model can get the right answer for the wrong reason. This is dangerous because the model may apply the same faulty reasoning to a different problem and confidently give the wrong answer. To evaluate reasoning, the harness logs every run’s full trace, and the same three LLM judges review it against the diagnostic criteria for reasoning and tool use written by our engineers. These criteria don’t count toward the accuracy score, since there’s more than one valid path to a correct result, but are worth tracking since flawed reasoning demonstrates a lack of generalizable understanding.

Our initial results cover 15 models: Claude Opus 4.8, Claude Opus 5, Claude Opus 5.5, Claude Sonnet 5.5, and Claude Fable 5.1 from Anthropic; GPT-5.6 Sol, GPT-6 Astra, GPT-6 Luna, GPT-6 Sol, and GPT-6.1 Sol from OpenAI; Grok 4.5, Grok 4.6, and Grok 4.7 from xAI; Kimi K3 from Moonshot; and GLM-5.3 from Zhipu AI (Z.ai). We’ll continue to expand this list, including new models as they’re released.

Every model currently runs in the same harness, Atlas, with the same tools, prompts, input files, and limits, and no per-model tuning. Atlas logs every trace itself instead of relying on the model provider, so traces are recorded the same way for every model. Future releases will compare models across other harnesses.

Each run uses the following configuration:

• Reasoning effort: High
• Model call limit: 600 calls per run
• Timeout: 3 hours per run
• Inactivity timeout: 45 minutes with no activity from model or tool call
• Context limit: runs that fill their context window are compacted and continue, with no scoring penalty
• Retries: runs stopped by an infrastructure issue, such as a license failure, are re-run; runs that hit a limit because of the model’s own behavior are scored as is

• More jobs, models, and harnesses: We will add new jobs, evaluate new models as they’re released, and compare results across other harnesses. Anyone can submit a job they would like evaluated: submissions will be ranked on our public leaderboard, and we will prioritize building evaluations for the most requested jobs.
• End-to-end jobs: In addition to individual tasks, we’ll also give models the full job at once. Today, each task starts from our engineers’ golden answer for the previous one. For a model to perform end-to-end hardware engineering work, it needs to build on its own earlier work and notice when it’s necessary to return to an earlier step, the way an engineer would.
• Context rot study: We’ll measure how performance degrades as a run’s context grows, building on Chroma’s research. Hardware runs are long and produce large tool outputs, such as simulation logs and layout files, while exact requirements from the start of a task need to hold at the end. Losing a single value, like a board thickness, can make a design fail.
• Calibrated diagnostics: We’ll calibrate the reasoning and tool-use criteria against our engineers’ grades. Judging whether a model’s engineering reasoning is sound requires domain expertise, so calibration will let us report diagnostic results with the same confidence as accuracy scores.
• Hallucination detection in traces: We’ll use an agent to review every run’s trace and flag claims that aren’t backed by a tool output or input file, such as invented values, simulations reported but never run, files cited but never opened, or results that don’t match what the tool returned. Our engineers will also spot-check a sample to measure how accurate the agent’s flags are.