Designing a 2.45 GHz Antenna: Evaluating 15 Models on HW-Bench’s First Job

Decomposing the job

The first job we evaluated in HW-Bench was designing a 2.45 GHz ISM band antenna for a telemetry link that reaches more than 50 m on a standard circuit board. It’s a relatively simple job, involving a single-band antenna on a 2-layer FR4 board, but a useful first evaluation because it (1) covers design, simulation, layout, and test planning, (2) requires driving tools like HFSS and KiCad, and (3) can be graded both by verifiable checks and by LLM judges for decisions that take engineering judgment.

The first step in building a job is breaking it down into the tasks an engineer would complete. Three RF engineers decomposed this job, built its golden answers, and calibrated the judges:

• Yang Su: RF engineer with over a decade of experience across Qualcomm and Meta. At Qualcomm, he delivered multiple generations of cellular reference designs from 3G to 5G, including one of the first commercially deployed phased-array antenna modules for 5G mmWave small cells.
• Nhat Tran: RF engineer who previously worked on mmWave systems-in-package at Keysight Technologies, with a Ph.D. from UC Davis on integrated circuits and antennas for mmWave and sub-terahertz transmitters.
• Zhaoji Fang: RF engineer with a Ph.D. from Brown in electrical and computer engineering, where his research focused on mmWave and terahertz wireless links, beam-steerable antennas, and programmable wavefront control.

With guidance from:

• Harish Krishnaswamy: Chief Scientist at Arena Physica. Harish is one of the world’s leading researchers in RF and mmWave phased arrays. He is also a professor at Columbia and co-founded mixComm, which Sivers Semiconductors acquired for its beamforming silicon. He holds a B.Tech. in Electrical Engineering from IIT Madras and a Ph.D. in Electrical Engineering from USC.
• Arun Natarajan: Distinguished Electrical Engineer at Arena Physica. Arun is the former CTO of Sivers Semiconductors and a professor at Yale. He built the first integrated phased-array transmitter and transceiver in silicon and has led several DARPA programs as principal investigator. He holds a B.Tech. in Electrical Engineering from IIT Madras and an M.S. and a Ph.D. in Electrical Engineering from Caltech.

It’s worth noting that hardware work is rarely linear. Once the board is laid out, an RF engineer re-simulates it. If it slightly misses spec, the engineer adjusts the layout. If it significantly misses, the engineer goes back to the simulator to retune the design. Our engineers estimate they typically iterate 3 to 15 times on a job like this. For this evaluation, each task runs separately and starts from the golden answer from the previous task, so the models aren’t tested on these loops. End-to-end jobs, where models work through the full job and have to catch and fix their own misses, are next on our roadmap.

Building the evaluation

Each task has five parts: a prompt, input files, tools, a golden answer, and a rubric. Here’s what they look like for Task 4, where the model turns a simulated antenna into a board a factory can build.

Prompt

Like every prompt in HW-Bench, our engineers wrote this one the way they would assign a task to a colleague. In about 200 words, it covers the requirements and where to save the output, and trusts the model, as they would a colleague, to choose the approach and tools. See our methodology for more on how we write prompts.

Create a fabrication-ready PCB layout in KiCad for the antenna in T3_project_output/: a 2-layer board on standard 1.6 mm FR4 that meets fabrication_capability_rev4p2.pdf. Where a fabrication rule conflicts with the antenna model, modify only the conflicting dimensions with the minimum changes needed to meet the rule. Configure the fabrication constraints as design rules in KiCad, and check the final board with DRC.

Deliver an outputs/ folder containing:

1. cad/: the complete KiCad project, including its design rules, that opens with nothing added.
2. fab/: Gerber X2 and Excellon drill files exported from the final board.
3. drc/: the DRC report (JSON) from the final board.
4. T4_output.md: a short summary stating whether the board is ready for fabrication, with any remaining DRC errors and warnings and each DRC exclusion with its reason, or that the task is inconclusive.

Input Files

Similar to the prompt, the input files are what our engineers would hand a colleague to start the task: the native project files and documents the work depends on. For Task 4, that’s:

• The retuned antenna from Task 3: This is the golden answer for Task 3, an HFSS project of a meandered inverted-F antenna (MIFA) retuned to 2.45 GHz on FR4, with its return loss and far-field data and plots. The board is 30.3 mm x 50.68 mm, with a 25 mm ground, the antenna 1.0 mm from the left edge, and four vias with a 0.400 mm drill.
• The fab house’s capability statement (fabrication_capability_rev4p2.pdf): the factory’s rules for conductors, drills, solder mask and silkscreen, laminate, board outline, and file formats.

Tools

The model also works directly in the same tools our engineers would use for the task. For Task 4, that’s KiCad, to lay out the board and set the fab house’s rules as design rules, with kicad-cli to run the design-rule check and export the Gerber and drill files, and Ansys HFSS, to check the layout against the original antenna design. Every model connects to these tools through the same integrations in Atlas, our harness, so differences in results come from the model, not the tool setup.

Golden Answer

To set the standard each model is graded against, our engineers completed every task themselves, with the same input files and tools the model gets. For Task 4, they laid out the Task 3 antenna on a 2-layer, 1.6 mm FR4 board.

For Task 4, the input antenna from Task 3 breaks two of the factory’s rules. Finding and fixing these is part of Task 4. In their golden answer, our engineers made the following adjustments:

• Copper to board edge: the copper runs right to the edge on the left, right, and bottom, where the fab house requires 0.250 mm. Our engineers extended the board outline by 0.250 mm on those three sides, to 30.8 mm x 50.93 mm, and left the copper and the top edge unchanged.
• Via annular ring: the rings are 0.050 mm, against a 0.075 mm minimum.

Apart from these two fixes, the golden answer matches the Task 3 model within ±50 µm, allowing for small rounding differences when moving the design from HFSS to KiCad. To pass, a model has to find both conflicts and fix each with the smallest change that meets the rule, without touching copper that would change how the antenna performs. The golden answer includes the board’s Gerber and drill files and a clean design-rule check report.

Rubric

Each task has a rubric written by our engineers to score accuracy. Each criterion is weighted by importance and graded either by a verifiable check in code or by an LLM judge (see our methodology for how criteria are weighted and how the judges are calibrated).

Task 4’s rubric has ten accuracy criteria:

The rubric also includes diagnostic criteria, which are reviewed in each run’s trace but don’t count toward the accuracy score. A few examples and why they matter:

• Reasoning (graded by a judge): the model reads every constraint in the fab house’s rules with its limit, and where the rules give more than one value, picks the one for this board: standard class, a routed edge (0.250 mm, not 0.300 mm for V-scored), and 1 oz copper (0.100 mm trace, not 0.150 mm). Clearances are applied as drawn, without stacking the ±0.150 mm routing tolerance on top. This shows whether a model read the rules for this specific board or applied a generic value, which the accuracy score alone wouldn’t reveal.
• Tool use (graded by a verifiable check): the design-rule check ran to completion on the final board. This separates a model that couldn’t operate KiCad from one that could operate it but produced a flawed layout.

Evaluating the models

Setup

Every model ran each task five times in the same harness, Atlas, with HFSS, KiCad, and Python connected the same way for every model. Each run was scored separately, and its cost and time were recorded alongside its accuracy. Costs are publicly listed prices on October 29, 2026. All runs used the following configuration:

• Reasoning effort: High
• Model call limit: 600 calls per run
• Timeout: 3 hours per run
• Inactivity timeout: 45 minutes with no activity from model or tool call
• Context limit: runs that fill their context window are compacted and continue, with no scoring penalty
• Retries: runs stopped by an infrastructure issue, such as a license failure, are re-run; runs that hit a limit because of the model’s own behavior are scored as is

Grading

As mentioned above, verifiable criteria were checked in code against each run’s output files. Judgment-based criteria were scored by an ensemble of three LLM judges: Claude Opus 5.5, GPT-6 Sol, and Grok 4.7. For this job, our engineers independently graded a sample of runs without seeing the judges’ scores, and we tuned the judges’ thresholds until they agreed with the engineers at least 90% of the time.

Analyzing the results

• No single model is best across a job. Routing each task to its best-performing model delivers higher accuracy at lower cost. The most accurate model varied by task: six different models each achieved the highest mean accuracy on at least one of the six tasks. If you used the single most accurate model, GPT-6 Astra, for all six tasks, the job would score 0.805 and take 88 minutes per run, at a cost of $50.49. We compared three routing strategies against it:
  • Model with top accuracy score per task: 0.875 accuracy at $13.66, 73% less than Astra, in 98 minutes, 12% more than Astra
  • Cheapest model within 5% of each task’s top accuracy score: 0.867 accuracy at $12.40, 75% less than Astra, also in 98 minutes
  • Fastest model within 5% of each task’s top accuracy score: 0.862 accuracy in 82 minutes, 7% faster than Astra, at $26.80, 47% less than Astra

• By mean accuracy, GPT-6 Astra and GPT-6 Sol were the only two models on both the cost and latency Pareto frontiers. GPT-6 Astra had the highest mean accuracy (0.805) and GPT-6 Sol was the fastest of all models (53 minutes). GPT-6.1 Sol, which is only on the cost frontier, came closest to Astra, as its 95% confidence interval for accuracy overlaps with Astra’s, so the difference is not statistically significant, at 17% of the cost ($8.42 vs. $50.49 per job run), though it took about 17 minutes longer.
  • Claude Opus 5.5 sits just off both frontiers. On cost, GPT-6.1 Sol has a higher mean accuracy (0.781) at half the price ($8.42). On time, GPT-5.6 Sol has a slightly higher mean accuracy (0.734) and finishes faster (67 minutes).
  • When run-to-run variation is taken into account, GPT-6.1 Sol, Opus 5.5, Fable 5.1, and Sonnet 5.5 could also be on the latency frontier, and GPT-5.6 Sol, Opus 5.5, and Opus 5 could also be on the cost frontier.

• Mean accuracy overstates reliability. Pass^k (the probability that k random runs of a task pass every critical criterion) averaged across tasks is a better measure. Pass⁵ is 0% for 11 of 15 models. Only four models had at least one task where all five runs passed. GPT-6 Astra is the highest at 67%, with all five runs passing on four of the six tasks. GPT-6.1 Sol achieved this on two tasks (33%) while Opus 4.8 and GPT-6 Luna only managed it on one task each (17%).
  • Three of the six models on the cost or latency frontier, GPT-6 Sol, GPT-5.6 Sol, and Sonnet 5.5, never passed every critical criterion on all five runs of any task. GPT-6 Sol is on both frontiers, yet only 37% of its runs passed every critical criterion.
  • Passing runs came from only a few models. On Task 3, GPT-6 Astra and GPT-6.1 Sol passed all five of their runs. Together they account for nearly half of the task’s 21 passing runs. On Tasks 4 and 5, every passing run came from an Anthropic model, and OpenAI models passed none of their 50 runs.

• Models are not yet reliable on tasks that require driving simulation and EDA tools. On the three tasks that required them (Tasks 3, 4, and 5), only 30 of 225 runs (13%) passed every critical criterion, compared with 92 of 225 (41%) on the text- and code-based tasks. On Task 4 (Translate to layout), only 3 of 75 runs passed every critical criterion. Most runs set the wrong board stackup: only 11 of 75 used the specified two layers of 1 oz copper on 1.6 mm FR4, and 8 of those 11 still failed another critical criterion. On Task 5 (Re-simulate & check), only 6 of 75 runs passed.

• Runs that make more tool calls don’t necessarily yield better results on tool-driven tasks. A tool call is one action the agent takes in its environment, such as reading a file, running code, or a step in HFSS or KiCad. On two of the writing tasks (Tasks 1 and 6), more tool calls were associated with higher accuracy: each doubling of tool calls came with a 0.11 (p<0.001***) to 0.12 higher accuracy score (p=0.003**). On Task 4 (Translate to layout) and Task 5 (Re-simulate and check), runs with more tool calls took longer, cost more, and were less accurate.

• Open-weight models still trailed closed models on accuracy and cost. GLM-5.3 and Kimi K3 ranked 12th and 15th of 15 by mean accuracy, yet cost $18.16 and $26.18 per job run, respectively. That’s more than twice the $8.42 of GPT-6.1 Sol, which had the second-highest accuracy at 0.781. Part of the gap may reflect age, since the open-weight models were released first, but GLM-5.3, which came out 25 days after Claude Opus 5 and cost about the same per Task 5 run ($4.87 vs. $5.24), reached only 89% of Opus 5’s accuracy on Task 5 and passed none of its runs, while Opus 5 passed three of five.
  • Open-weight models also rarely passed every critical criterion. GLM-5.3 passed 5 of its 30 runs and Kimi K3 passed 2, compared with 17 for GPT-6.1 Sol. Neither open-weight model passed a single run on Tasks 1, 4, 5, or 6, or passed all five runs of any task.

Acknowledgements

Thank you to Pratap Ranade for conceiving HW-Bench; Heidi Kim and Shyama Dave for leading the program; Arya Hezarkhani for technical advisory; Claire Pan and Christopher Bryant for writing; the Arena Physica RF engineering team for their expertise; and the Arena Physica engineering team for building the evaluation infrastructure that HW-Bench runs on.