The next great leap for humanity won’t happen in software alone
It will happen in the physical world, in the chips, machines, and systems that run modern life. Fusion reactors will power entire cities. Rockets will carry the first colonists to Mars. Medical scanners will detect cancer years sooner. For decades, we’ve accepted that these breakthroughs were generations away, bottlenecked by a slow hardware design loop. AI is changing that, and we’re already seeing evidence of this with Atlas, our harness for hardware engineering, and Heaviside, our physics-native foundation model for electromagnetism.
For most hardware leaders, the question is no longer whether AI can help, but how. From fast-rising startups to the Fortune 500, we hear the same three concerns:
- Value: It’s unclear where else AI is creating real production-level impact
- Cost: Token bills are climbing with no definitive ROI
- Security: Their most sensitive IP is flowing to a handful of external model providers
Addressing these concerns requires understanding (1) task-level accuracy, showing where each model excels and where it falls short, (2) cost and time spent per task, and (3) comparative performance of externally managed versus locally hosted models, showing whether a secure model they fully own can perform sufficiently well for their use case.
Historically, when the same questions came up in the software industry, benchmarks built on real coding tasks gave teams a clear way to compare models and decide which work to hand to AI. So why doesn’t a standard benchmark exist for hardware?
Effectively evaluating hardware engineering work is difficult
Hardware context is scattered across modalities and formats. Software work primarily lives in text-based documents, spreadsheets, and git repos. Hardware design lives in 3D geometries, schematics, and layouts, fragmented across specialized formats and tools. It also demands far more context, and from far more places: a software task usually fits in one repo, while a hardware task pulls datasheets, models, and files for every component and connection, scattered across different tools. The few hardware benchmarks that exist each cover one slice of the work (e.g., PCB layout, 3D CAD modeling) rather than the end-to-end hardware lifecycle.
Hardware has no fast verifier, so evaluating correctness is difficult and slow. Software has unit tests that can check code in milliseconds. While hardware is governed by physics (so most engineering work has a right answer), there are only two ways to verify whether a design works: in simulation or in the real world. Electromagnetic, thermal, and circuit simulations can take hours or days, while physical testing requires building and instrumenting hardware in a lab where real interference, manufacturing tolerances, and equipment error introduce variables that don’t exist in simulation. Neither provides the fast, repeatable feedback loop that software gets from running a test suite.
In hardware, the most valuable data is private. Coding benchmarks can draw on millions of public repositories, but production hardware designs are among the most closely guarded IP in the world. The same is true of the data that captures how those designs perform in practice. Without access to real production designs and their test results, hardware evaluations tend to fall back on textbook problems and academic designs that poorly represent the complexity of real engineering work.
Measurement has always been core to our work at Arena Physica: the very first version of Atlas shipped with our internal framework for evaluating frontier models on real hardware engineering tasks. We used it not only to track our own progress, but to guide our customers on where their time and money would go furthest. Today, we’re releasing that framework to the world.
Introducing HW-Bench, the first hardware engineering benchmark built with the world’s most advanced hardware companies
HW-Bench is a rigorous, hardware-native evaluation suite. Our founding partners build some of the most advanced hardware in the world, from the chips behind modern supercomputing to the autonomous systems and vessels redefining defense at sea and beyond. More on them coming soon.
HW-Bench uniquely addresses the challenges of building a realistic hardware benchmark:
- It captures the full hardware context. Every task includes the input files an engineer would use (e.g., schematics, layouts, simulation projects, and datasheets), in their native formats. Models also use the same tools engineers do (e.g., Ansys HFSS, KiCad). This enables HW-Bench to cover the full scope of hardware work, across a system’s mechanical, electrical, and firmware components.
- Every answer has a ground truth. Our engineers complete each task to create a golden answer, and subject matter experts from our partners review every task and rubric. Models are then graded against that golden answer. Outputs with a right answer are checked in code. Outputs that require subjective evaluation are graded by an ensemble of LLM judges calibrated by our engineers. Some designs are fabricated and tested in our lab to confirm the simulations match real-world performance.
- It draws on production data. Our partners contribute data from their production hardware: fabricated designs, lab test data, and field performance results. The data stays private to them, but it gives HW-Bench access to the true complexity of hardware engineering.
HW-Bench evaluates models on complete jobs, not just isolated tasks, quantifying their actual hardware engineering work
We started by mapping the universe of hardware engineering tasks into a grid, organized by discipline and stage.
Mapping the universe of hardware engineering work
- A cell
- one discipline against one stage of the life cycle
- A job
- within a cell, a unit of useful work
- A task
- one of the steps a job is decomposed into
Each cell in the grid includes multiple jobs (the smallest complete unit of work an engineer hands off as a finished deliverable) and each job is decomposed into tasks, replicating exactly how an engineer would do the work.
For each task, we build an evaluation with five parts: (1) a prompt written the way an engineer would hand to an agent, (2) the input files in their native formats, (3) the tools an engineer would use, such as KiCad or Ansys HFSS, (4) a golden answer from an expert, and (5) a rubric, written with the expert, which scores the accuracy of the output.
Since hardware is governed by physics, most engineering work can be verified against a clear answer: an antenna either meets its return-loss spec across the target band or it doesn’t. HW-Bench grades model outputs the same way, using verifiable checks wherever there’s a right answer. When a criterion requires subjective judgment, such as whether a test plan covers the failure modes that matter, an LLM judge calibrated by human engineers determines the accuracy criteria.
Every task is run five times on each model and harness, and is measured on accuracy, cost, and latency. Our initial results use Atlas as the harness to call the following models: Claude Opus 4.8, Claude Opus 5, Claude Opus 5.5, Claude Sonnet 5.5, and Claude Fable 5.1 from Anthropic; GPT-5.6 Sol, GPT-6 Astra, GPT-6 Luna, GPT-6 Sol, and GPT-6.1 Sol from OpenAI; Grok 4.5, Grok 4.6, and Grok 4.7 from xAI; Kimi K3 from Moonshot; and GLM-5.3 from Zhipu AI (Z.ai).
Results from our first job show that choosing a model requires looking beyond a single score
We started with the relatively simple job of designing a 2.45 GHz ISM band antenna and decomposed the job into six tasks: (1) derive the spec, (2) select a reference design, (3) retune the reference design in HFSS, (4) translate the design to a PCB layout in KiCad, (5) re-simulate the layout and check it against the spec, and (6) write a test plan.
One job, six tasks
Our findings:
- No single model is best across a job. Routing each task to its best-performing model delivers higher accuracy at lower cost. The most accurate model varied by task: six different models each achieved the highest mean accuracy on at least one of the six tasks. If you used the single most accurate model, GPT-6 Astra, for all six tasks, the job would score 0.805 and take 88 minutes per run, at a cost of $50.49. We compared three routing strategies against it:
- Model with top accuracy score per task: 0.875 accuracy at $13.66, 73% less than Astra, in 98 minutes, 12% more than Astra
- Cheapest model within 5% of each task’s top accuracy score: 0.867 accuracy at $12.40, 75% less than Astra, also in 98 minutes
- Fastest model within 5% of each task’s top accuracy score: 0.862 accuracy in 82 minutes, 7% faster than Astra, at $26.80, 47% less than Astra
No single model topped every task
Accuracy, cost and latency per task for 15 models. 6 different models were most accurate on at least one task.
Anthropic
Claude Opus 5.5Claude Opus 5Claude Fable 5.1Claude Opus 4.8Claude Sonnet 5.5OpenAI
GPT-6 AstraGPT-6.1 SolGPT-5.6 SolGPT-6 SolGPT-6 LunaxAI
Grok 4.7Grok 4.6Grok 4.5Moonshot (open weight)
Kimi K3Z.ai (open weight)
GLM-5.3Select a model in the key to highlight it on all three radars.
Notes
Each axis is one task. Accuracy: mean score of the model’s 5 runs (0 to 1). Cost: mean list-price cost per run (USD, prices as of Oct 29, 2026). Latency: mean wall-clock time per run. Cost and latency axes point outward = cheaper / faster.
- By mean accuracy, GPT-6 Astra and GPT-6 Sol were the only two models on both the cost and latency Pareto frontiers. GPT-6 Astra had the highest mean accuracy (0.805) and GPT-6 Sol was the fastest of all models (53 minutes). GPT-6.1 Sol, which is only on the cost frontier, came closest to Astra, as its 95% confidence interval for accuracy overlaps with Astra’s, so the difference is not statistically significant, at 17% of the cost ($8.42 vs. $50.49 per job run), though it took about 17 minutes longer.
- Claude Opus 5.5 sits just off both frontiers. On cost, GPT-6.1 Sol has a higher mean accuracy (0.781) at half the price ($8.42). On time, GPT-5.6 Sol has a slightly higher mean accuracy (0.734) and finishes faster (67 minutes).
- When run-to-run variation is taken into account, GPT-6.1 Sol, Opus 5.5, Fable 5.1, and Sonnet 5.5 could also be on the latency frontier, and GPT-5.6 Sol, Opus 5.5, and Opus 5 could also be on the cost frontier.
Accuracy vs cost and latency
Notes
Each dot is one model. Accuracy: mean of the six task means (0 to 1). Cost: sum of the six per-task mean costs per run, at list prices as of Oct 29, 2026. Time: sum of the six per-task mean wall-clock times. Frontier: models that no other model beats on both accuracy and cost (or time). Error bars and shaded band: 95% bootstrap intervals, resampling each model’s 5 runs per task 1,000 times. Blue dots: error bar reaches the band. Cost and time axes are log scale.
- Mean accuracy overstates reliability. Passk (the probability that k random runs of a task pass every critical criterion) averaged across tasks is a better measure. Pass5 is 0% for 11 of 15 models. Only four models had at least one task where all five runs passed. GPT-6 Astra is the highest at 67%, with all five runs passing on four of the six tasks. GPT-6.1 Sol achieved this on two tasks (33%) while Opus 4.8 and GPT-6 Luna only managed it on one task each (17%).
- Three of the six models on the cost or latency frontier, GPT-6 Sol, GPT-5.6 Sol, and Sonnet 5.5, never passed every critical criterion on all five runs of any task. GPT-6 Sol is on both frontiers, yet only 37% of its runs passed every critical criterion.
- Passing runs came from only a few models. On Task 3, GPT-6 Astra and GPT-6.1 Sol passed all five of their runs. Together they account for nearly half of the task’s 21 passing runs. On Tasks 4 and 5, every passing run came from an Anthropic model, and OpenAI models passed none of their 50 runs.
Mean accuracy overstates reliability
Pass1 to pass5 for each model, averaged over Tasks 1 to 6 with 5 runs per task. Pass5 stays above 0% for 4 of 15 models.
Anthropic
Claude Opus 5.5Claude Opus 5Claude Fable 5.1Claude Opus 4.8Claude Sonnet 5.5OpenAI
GPT-6 AstraGPT-6.1 SolGPT-5.6 SolGPT-6 SolGPT-6 LunaxAI
Grok 4.7Grok 4.6Grok 4.5Moonshot (open weight)
Kimi K3Z.ai (open weight)
GLM-5.3Select a line or a name for its pass1 to pass5.
Notes
Every task is run 5 times per model in the same harness. A run passes if it passes every critical criterion. Passk: the chance that k random runs (of 5) of a task all pass every critical criterion. Lines average passk over Tasks 1 to 6. Blue: pass5 above 0%.
- Models are not yet reliable on tasks that require driving simulation and EDA tools. On the three tasks that required them (Tasks 3, 4, and 5), only 30 of 225 runs (13%) passed every critical criterion, compared with 92 of 225 (41%) on the text- and code-based tasks. On Task 4 (Translate to layout), only 3 of 75 runs passed every critical criterion. Most runs set the wrong board stackup: only 11 of 75 used the specified two layers of 1 oz copper on 1.6 mm FR4, and 8 of those 11 still failed another critical criterion. On Task 5 (Re-simulate & check), only 6 of 75 runs passed.
Pass1 to pass5 for each task
Averaged over 15 models, 5 runs each.
Task 2 Select reference design 55% → 20%Task 3 Retune reference design | Ansys HFSS 28% → 13%Task 6 Write test plan 32% → 13%Task 1 Derive spec 36% → 7%Task 4 Translate to layout | KiCad, Ansys HFSS 4% → 0%Task 5 Re-simulate & check | Ansys HFSS, KiCad 8% → 0%
| Task | pass1 | pass2 | pass3 | pass4 | pass5 |
|---|---|---|---|---|---|
| Task 1 Derive spec | 36% | 19% | 13% | 9% | 7% |
| Task 2 Select reference design | 55% | 40% | 31% | 25% | 20% |
| Task 3 Retune reference design (Ansys HFSS) | 28% | 17% | 14% | 13% | 13% |
| Task 4 Translate to layout (KiCad, Ansys HFSS) | 4% | 1% | 0% | 0% | 0% |
| Task 5 Re-simulate & check (Ansys HFSS, KiCad) | 8% | 3% | 1% | 0% | 0% |
| Task 6 Write test plan | 32% | 20% | 15% | 13% | 13% |
Notes
Passk: the chance that k random runs (of 5) of a task all pass every critical criterion, averaged over the 15 models. Blue: Tasks 1, 2 and 6. Orange: Tasks 3, 4 and 5 (Ansys HFSS, KiCad).
- Runs that make more tool calls don’t necessarily yield better results on tool-driven tasks. A tool call is one action the agent takes in its environment, such as reading a file, running code, or a step in HFSS or KiCad. On two of the writing tasks (Tasks 1 and 6), more tool calls were associated with higher accuracy: each doubling of tool calls came with a 0.11 (p<0.001***) to 0.12 higher accuracy score (p=0.003**). On Task 4 (Translate to layout) and Task 5 (Re-simulate and check), runs with more tool calls took longer, cost more, and were less accurate.
Each dot is one run
Tool calls (log scale) against accuracy, one dot per run.
Notes
Each dot is one run: x = tool calls in the run (log scale), y = run accuracy (0 to 1). A tool call is one action the agent takes, such as reading a file, running code, or a step in HFSS or KiCad. Line: least-squares fit of accuracy on log₂(tool calls). Slope: the change in accuracy each time tool calls double. r: Pearson correlation coefficient between accuracy and log₂(tool calls) (0 = no linear relationship, ±1 = every dot on the line). p: two-sided p-value for the slope, i.e. the chance of a slope at least this steep if accuracy did not depend on tool calls.
- Open-weight models still trailed closed models on accuracy and cost. GLM-5.3 and Kimi K3 ranked 12th and 15th of 15 by mean accuracy, yet cost $18.16 and $26.18 per job run, respectively. That’s more than twice the $8.42 of GPT-6.1 Sol, which had the second-highest accuracy at 0.781. Part of the gap may reflect age, since the open-weight models were released first, but GLM-5.3, which came out 25 days after Claude Opus 5 and cost about the same per Task 5 run ($4.87 vs. $5.24), reached only 89% of Opus 5’s accuracy on Task 5 and passed none of its runs, while Opus 5 passed three of five.
- Open-weight models also rarely passed every critical criterion. GLM-5.3 passed 5 of its 30 runs and Kimi K3 passed 2, compared with 17 for GPT-6.1 Sol. Neither open-weight model passed a single run on Tasks 1, 4, 5, or 6, or passed all five runs of any task.
Best open-weight model vs the top frontier model on each task
Mean accuracy, pass1 and days between releases, 5 runs per task.
Notes
For each task: the best open-weight model (by mean accuracy) against the top model, the model with the highest mean accuracy on the task. Mean accuracy: mean of the model’s 5 runs (0 to 1). Pass1: share of the 5 runs that passed every critical criterion. Days between releases: the open-weight model’s release date minus the top model’s; below 0, the open-weight model came out first. Colored dot: open-weight model, in its lab’s color. White dot: top model.
Note on Task 3 run times
Our retune task asks the model to run only the narrowband sweep, which tunes the antenna’s resonant frequency. It sets up the wideband sweep, which checks for additional resonances, but doesn’t run it. A full retune with both sweeps takes longer (mostly solver time), so the times we report for Task 3 are not representative of the real work done by an engineer.
The 2.45 GHz antenna is just the beginning. We’re now expanding HW-Bench across engineering disciplines and from single components to full systems, including the design, manufacture, and test of a 140 GHz, 512-element D-band phased array.
Join us in building HW-Bench
Whether you’re an engineer who wants to help shape the framework, a company with designs or data to contribute, a researcher who wants to build on HW-Bench or someone curious about how frontier models perform on real hardware work, sign up for updates below or reach out to us at [email protected].