The next great leap for humanity won’t happen in software alone
It will happen in the physical world, in the chips, machines, and systems that run modern life. Fusion reactors will power entire cities. Rockets will carry the first colonists to Mars. Medical scanners will detect cancer years sooner. For decades, the industry has accepted that these breakthroughs were generations away, bottlenecked by a slow hardware design loop. AI is changing that, and we're already seeing evidence of this with Atlas, our harness for hardware engineering, and Heaviside, our physics-native foundation model for electromagnetism.
For most hardware leaders, the question is no longer whether AI can help, but where. From fast-rising startups to the Fortune 500, we hear the same three concerns: (1) it’s unclear where AI is creating real production-level impact, (2) token costs are climbing with no definitive ROI, and (3) their most sensitive IP is flowing to a handful of model providers.
Hardware engineering needs a standard
A hardware engineering benchmark should give users visibility into the decisions that matter the most: (1) task-level accuracy, showing where each model performs well and where it falls short, (2) cost and time per task, by model, and (3) performance of managed versus locally hosted models, showing whether a model you fully own and keeps your IP secure, performs sufficiently well for your use case.
Today's frontier AI models are evaluated on benchmarks for domains like math and software, but effectively evaluating proficiency in hardware engineering poses some unique challenges:
Hardware context is scattered across formats. Software, legal, and accounting work primarily live in text-based documents, spreadsheets, and git repos. Hardware design lives in 3D geometries, schematics, and layouts, fragmented across specialized formats and tools. It also demands far more context, and from far more places: a software task usually fits in one repo, while a hardware task pulls datasheets, models, and files for every component and connection, scattered across different tools. The few hardware benchmarks that exist each cover one slice of the work (i.e., PCB layout, 3D CAD modeling) rather than the end-to-end hardware lifecycle.
Hardware has no fast verifier, so evaluating correctness is difficult and slow. Software has unit tests that check code in milliseconds. Hardware has two ways to verify whether a design works: simulation or the real world. Electromagnetic, thermal, and circuit simulations can take hours or days, while physical testing requires building and instrumenting hardware in a lab, where real interference, manufacturing tolerances, and equipment error introduce variables that don’t exist in simulation. Neither provides the fast, repeatable feedback loop that software gets from running a test suite.
Hardware’s most valuable data is private. Coding benchmarks can draw on millions of public repositories, but production hardware designs are among the most closely guarded IP in the world. The same is true of the data that captures how those designs perform in practice. Without access to real production designs and their test results, hardware evaluations tend to fall back on textbook problems and academic designs that poorly represent the complexity of real engineering work.
Measurement has always been core to our work at Arena Physica: the very first version of Atlas shipped with Litmus, an internal framework for evaluating frontier models on real hardware engineering tasks. We used Litmus not only to track our own progress, but to guide our customers on where their time and money would go furthest, including where AI still falls short. Today, we're releasing that framework to the world.
Introducing HW-Bench, the first hardware engineering benchmark built with the world’s most advanced hardware companies
HW-Bench is a rigorous, hardware-native evaluation suite. Our founding partners include AMD, whose chips power the world's fastest supercomputers; Anduril, whose autonomous systems are redefining modern defense; and SpaceX, whose rockets carry most of the world's payload to orbit. Together, they're building some of the most complex hardware on Earth and beyond.
HW-Bench has two layers. The first is a set of realistic engineering tasks, built by our engineers and reviewed by our partners. We'll release a subset of this data publicly, including the schematics, tasks, and results. The second layer draws on our partners' production hardware including fabricated designs, test data, and field performance results. Together, we curate the evaluations, review every result, and share our summarized findings.
The HW-Bench methodology
HW-Bench starts with a map of all hardware engineering jobs, organized by discipline and stage, from design to build to sustainment. Each cell on the map represents a job, the smallest complete piece of work an engineer hands off as a finished deliverable, and we break every job into the steps that constitute it, replicating exactly how an engineer actually does the job.
We start by mapping the universe of hardware engineering work
The hardware development life cycle is fragmented by stage and by discipline.
- A cell
- one discipline against one stage of the life cycle
- A task
- within a cell, a unit of useful work
- An atomic step
- the steps that a task is made of
The antenna task, as its six atomic steps
- E1Derive spec
- E2Choose ref
- E3Retune
- E4Draw board
- E5Re-import
- E6Test plan
Every step becomes an evaluation with four parts: the input files, the tools an engineer would use, a golden answer from an expert, and a rubric that scores the model's output. The rubric applies deterministic checks where the output has a right answer, and a calibrated LLM judge where evaluation requires judgment.
Every task is run five times per model, in the same harness, and scored for accuracy, cost, and latency. Models evaluated in our initial results include Claude Opus 4.8, Claude Opus 5, Claude Opus 5.5, Claude Sonnet 5.5, and Claude Fable 5.1 from Anthropic; GPT-6.1 Sol, GPT-6 Astra and GPT-6 Sol from OpenAI; Grok 4.5, Grok 4.6 and Grok 4.7 from xAI; Kimi K3 from Moonshot; and GLM-5.3 from Zhipu AI (Z.ai).
The result shows not just how capable a model is, but whether it's worth using, and on which tasks. Read a deep dive on our methodology here.
The first job: designing a 2.45 GHz antenna
The first job we built evals for was designing a 2.45 GHz ISM band antenna, broken into six steps: writing the spec, picking a reference design, retuning it in HFSS, drawing the board in KiCad, verifying the result, and writing the test plan.
One hardware job is six steps across four tools
88 engineer-hours end to end, including one retune loop from E5 back to E3.
Sources
Steps, tools, roles and hours from the task breakdown in Introducing HW-Bench. Best model and score per step from the same paper's results: mean accuracy over five runs, same harness.
- Claude Opus 5.5 and Fable 5.1 had the highest average accuracy scores within 2 points of each other, but no single model topped every step. GPT-6 Astra wrote the best spec, Grok 4.6 picked the best reference design, and Kimi K3 came within 2% of the best retune.
No single model topped every step
Opus 5.5 was most accurate on 3 of 6 steps; 4 different models won at least one.
Sources
Every step is run 5 times per model in the same harness. Cost and latency are min-max scaled per step, so the edge is the cheapest or fastest model on that step.
Approximate: the per-model values on each axis were read off the draft figure in the PDF, not taken from the run data, and are good to about two points on the 0 to 1 scale. They will be replaced with the run data before launch.
- An ensemble of models matches the best single model at lower cost. Opus 5.5 scored highest on average across the six steps, but routing each step to the cheapest model within 5% of that step's best matches its accuracy at 17% lower cost, in the same time. Allowing scores within 10% cuts costs by a third and is 8% faster.
- On tasks that only require text and code-based work, accuracy scores converge across models, making cost and speed the deciding factors. The models on the Pareto frontier for these steps are within 9 points of each other, compared with 20 points on steps that drive HFSS and KiCad. Grok 4.7 scored 92 to 95% of the best models, at 17 to 30% of their cost and 52 to 69% of their time.
The frontier spreads out once simulators and EDA come in
Mean score, 0 to 1, against the time one pass takes. Best to worst on the frontier is 9 points on text and code steps, 20 once tools are involved.
Sources
Every step is run 5 times per model in the same harness. Each point is one model's mean accuracy over the three steps in its group, against the time one pass over those steps took.
Approximate: scores and times were read off the draft figure in the PDF, not taken from the run data, and are good to about two points on the 0 to 1 scale. They will be replaced with the run data before launch.
- Model choice matters most on tasks that drive workbench applications like Ansys HFSS and KiCad, where the top models lead by 17 to 25%. Claude Opus 5.5 and Fable 5.1 scored within 2 points of each other, but Fable 5.1 cost 3.6× more and finished about 20% sooner. Both scored 17 to 25% higher than Kimi K3, GLM-5.3 and Grok 4.6, and Opus 5.5 still cost less than Kimi K3 or GLM-5.3.
- One good run isn't a capability. Claude Opus 5.5 had the best average on the test plan, but its five runs on identical inputs ranged from 60% to 90%, and Kimi K3's verification runs ranged from 23% to 88%. A single strong score can hide large run-to-run swings.
Five identical runs can give very different scores
Re-import & verify is the most variable step, with a median spread of 0.10; choose a reference is the steadiest, at 0.03.
Sources
Every step is run 5 times per model in the same harness. Standard deviation of each model's five run accuracy scores on a step, on the 0 to 1 scale; the orange tick is the median across all models.
Approximate: each dot's position and the medians were read off the draft figure in the PDF, not taken from the run data, and are good to about two points on the 0 to 1 scale. They will be replaced with the run data before launch.
- A high score doesn't tell you what a model does when it's missing information. When a value the test plan depends on was moved, deleted or given two conflicting figures, Kimi K3, GLM-5.3 and GPT-6 Sol made up a number without warning in 27 to 47% of runs. Only Opus 5.5, Fable 5.1 and GPT-6 Astra never hid a missing value, and they were also the most expensive.
When a value went missing, four of seven models made one up
Step 6, the test plan. Only three never hid it; Kimi K3 invented a number with no warning in 7 of 15 runs.
Sources
Every ablation set up is run 5 times per model in the same harness. The three ablation setups include (1) value moved to another file (2) two conflicting values in the spec (3) value deleted. Silent = the judge found a number with no stated source, or no mention of the missing value.
Percentages are of 15 runs per model and are rounded, so a bar can sum to 101%. Two of the 105 runs were still awaiting the judge at publication; their tier comes from reading the output.
- Open-weight models are closing the gap. On picking a reference design, Kimi K3 tied Claude Opus 5 and Fable 5.1 at 85%, at a third of Opus 5's cost. It scored just ~3% behind the best model, Grok 4.6, at about a quarter of the cost and ~13% of the time.
Open-weight models are close on some steps, far behind on others
The best open-weight model is within 10% of the top model on 2 of 6 steps.
Sources
Every step is run 5 times per model in the same harness. Model pairs are the highest average accuracy score among SOTA models vs highest average accuracy score among open weight models. % of top and the gap between release dates as published.
The open-weight model's score is the top score times its published % of top, so it is good to that percentage's rounding.
Designing a 2.45 GHz ISM band antenna is just one task out of the many that HW-Bench covers. Read a deep dive on our initial findings here.
Join us in building HW-Bench
Our first full results go public next month. Whether you're an engineer who wants to help shape the framework, a company with designs or data to contribute, or a researcher who wants to build on HW-Bench, stay updated here or reach out to us at [email protected].