Deep dive: Method and results on designing an ISM band RF antenna

Our first eval was designing a 2.45 GHz ISM band antenna. This deep dive walks through each step of the HW-Bench methodology, from how an engineer actually does the job to what we saw after running the evals.

Selecting a task

The task was designing a 2.45 GHz ISM band antenna for a telemetry link that reaches more than 50m on a standard circuit board. The deliverables are a PCB layout ready to fabricate and verified against the spec, and a test plan ready for the bench. In the universe of hardware engineering tasks, it sits within electrical engineering, under the subdomain of RF, EM and Optical, in the Design and Simulate stage. See how we map that universe in our methodology.

Breaking the task down into steps

  • Input data: system requirements, FCC and EU regulations, chip datasheets, a library of past designs, and the board factory's manufacturing rules.
  • Tools: an EM simulator like HFSS, a PCB layout tool like KiCad, and Python scripts to drive the test instruments.

Here are the six steps:

Fig. 01 · One job, six steps, four tools

One hardware job is six steps across four tools

88 engineer-hours end to end, including one retune loop from E5 back to E3.

Text-basedNoneSimulatorAnsys HFSSEDAKiCad (Gerber)Code executionPython sandboxslight miss, retune neededDerive specE1 · 8hChoose refE2 · 12hRetuneE3 · 16hDraw boardE4 · 16hRe-importE5 · 20hTest planE6 · 16hTEST BENCHE1Derive the specText only · 8 hE2Choose a referenceText only · 12 hE3Retune in the solverAnsys HFSS · 16 hE4Draw the boardKiCad (Gerber) · 16 hE5Re-import & verifyAnsys HFSS · 20 hE6DVT test planPython sandbox · 16 hTEST BENCH
Select a step, or walk the job.
Sources

Steps, tools, roles and hours from the task breakdown in Introducing HW-Bench. Best model and score per step from the same paper's results: mean accuracy over five runs, same harness.

Each step is the smallest piece of the job an engineer hands off as finished. Select a step for who does it, how long it takes and which model scored best on it, or walk the job end to end. Illustrative.

Few tasks in hardware engineering are linear, including this one. For example, once the board is drawn, the RF engineer re-simulates it. If it slightly misses spec, this task returns to the layout design stage. If it significantly misses, this task returns to the simulator step for retuning.

Building the eval

Each step gets a prompt, input files in their native formats, the workbench tools an engineer would use, a golden answer produced by an engineer, and a rubric. Here's what that looks like for one step (E3) from the RF antenna job:

  • Prompt: Retuning the design chosen from the IP library in the EM simulator to meet the specs.
  • Input files: the design picked from the IP library in the previous step, as an HFSS project and build script, with its KiCad board and measured S11 file. Also the spec sheet, and the range each dimension is allowed to move within. In the step before this one, the model picks a starting point from an IP library we built in-house, by taking existing antenna designs, such as an open-source MIFA, and retuning them to different specs.
  • Workbench applications: Ansys HFSS to retune the design and Python interface to analyze results from the retuning and simulation.
  • Golden answer: Yang retuned the design in HFSS himself, solving for the feed position and sweeping the dimensions until it met every spec line.
  • Rubric: there's no single right geometry, so any design that meets every spec line with all dimensions in range passes. First, a pass/fail check that every dimension stayed inside its range. Then each spec line is checked against the model's own simulation results. The model also has to show how it got there, for example that it solved for the feed position rather than copying it across.

Grading

The rubric is scored with deterministic checks where there is a right answer, and a calibrated LLM judge where it takes judgment. Here's how the E3 rubric is graded:

  • Range check (deterministic): did every dimension stay inside its allowed range? If not, the design fails, however good it looks.
  • Spec lines (deterministic): centre frequency, bandwidth, return loss and gain are read from the model's own simulation results and checked against the spec.
  • How it got there (judge): did the model solve for the feed position rather than copy it across, and are its claims backed by its own solves?

Running the models

Every model runs in the same harness, with the same tools, prompt and input files, five times per atomic step.

For the RF antenna, we ran both SOTA and open-weight models on each of the six atomic steps. Each model (at high effort level) ran each step five times, in the same harness, with HFSS, KiCad and Python connected the same way for every model. Every run was scored separately, and we recorded its cost and time alongside its score.

Results

An ensemble approach for models is useful for long horizon, hardware workflows. Four different models topped the six steps. GPT-6 Astra wrote the best spec (0.77), Grok 4.6 picked the best reference design (0.88), Claude Opus 5 did the best retune (0.80), and Claude Opus 5.5 was best at the board (0.86), verification (0.89) and the test plan (0.78). Routing each step to the cheapest model within 5% of that step's best matches Claude Opus 5.5's accuracy at 17% lower cost, in the same time.

Fig. 02 · Best model per step

No single model topped every step

Opus 5.5 was most accurate on 3 of 6 steps; 4 different models won at least one.

Opus 5.5 · Mean accuracy, 0 to 1Best on the stepE1 Derive spec≈0.67GPT-6 Astra 0.77E2 Choose ref≈0.86Grok 4.6 0.88E3 Retune≈0.80Opus 5 0.80E4 Draw board≈0.86Opus 5.5 0.86E5 Re-import≈0.89Opus 5.5 0.89E6 Test plan≈0.78Opus 5.5 0.780.00.51.0
E1 Derive specbest GPT-6 Astra 0.77≈0.67E2 Choose refbest Grok 4.6 0.88≈0.86E3 Retunebest Opus 5 0.80≈0.80E4 Draw boardbest Opus 5.5 0.86≈0.86E5 Re-importbest Opus 5.5 0.89≈0.89E6 Test planbest Opus 5.5 0.78≈0.780.00.51.0
Sources

Every step is run 5 times per model in the same harness. Cost and latency are min-max scaled per step, so the edge is the cheapest or fastest model on that step.

Approximate: the per-model values on each axis were read off the draft figure in the PDF, not taken from the run data, and are good to about two points on the 0 to 1 scale. They will be replaced with the run data before launch.

One bar per step for the chosen model; orange where it was the best on that step. Switch between the three models that won steps, or show all 13, and no one model lights all six. Cost and latency redraw the same bars. Illustrative.

Model reliability is stochastic, one good run is not a capability. Claude Opus 5.5 has the best mean score on the test plan, but its five runs, from identical inputs, ranged from 0.60 to 0.90. Every one of the 13 models swung by more than 20 points on at least one step. The widest was Kimi K3 on verification, from 0.23 to 0.88 (SD 0.23).

Fig. 03 · Run-to-run spread

Five identical runs can give very different scores

Re-import & verify is the most variable step, with a median spread of 0.10; choose a reference is the steadiest, at 0.03.

0.000.050.100.150.200.25SD of the 5 run scoresless variablemore variableE1 Derive the specmedian 0.07E2 Choose a referencemedian 0.03E3 Retune in the solvermedian 0.06E4 Draw the boardmedian 0.04E5 Re-import & verifymedian 0.10E6 DVT test planmedian 0.09
0.000.100.20less variablemore variableE1 Derive the specmedian 0.07E2 Choose a referencemedian 0.03E3 Retune in the solvermedian 0.06E4 Draw the boardmedian 0.04E5 Re-import & verifymedian 0.10E6 DVT test planmedian 0.09
Sources

Every step is run 5 times per model in the same harness. Standard deviation of each model's five run accuracy scores on a step, on the 0 to 1 scale; the orange tick is the median across all models.

Approximate: each dot's position and the medians were read off the draft figure in the PDF, not taken from the run data, and are good to about two points on the 0 to 1 scale. They will be replaced with the run data before launch.

Each point is one model on one step: the standard deviation of its five run scores. The orange tick is the median across the 13 models. Select a dot for its value, or a lab in the key to pick out its models; re-sort to put the most variable steps first. Illustrative.

Model performance is most variable for tasks with workbench integrations. On the three text and code steps, every model except GPT-6 Sol scores within 12 points of the others, and 8 of 13 are within 10 points of the best. On the three steps that drive HFSS and KiCad, the same spread is 17 points, and only 5 models are within 10 points of the best.

Fig. 04 · Accuracy against time

The frontier spreads out once simulators and EDA come in

Mean score, 0 to 1, against the time one pass takes. Best to worst on the frontier is 9 points on text and code steps, 20 once tools are involved.

Text + code only (E1, E2, E6)
0.60.70.8010 min20 min30 min40 min50 minKimi K3Grok 4.7Sonnet 5.5Fable 5.1Opus 5.5
Simulator + EDA (E3, E4, E5)
0.60.70.801h2h3hGPT-6 SolOpus 4.8Sonnet 5.5Fable 5.1Opus 5.5
TEXT + CODE ONLYE1, E2, E69 ptsKimi K3 0.68Opus 5.5 0.77SIMULATOR + EDAE3, E4, E520 ptsGPT-6 Sol 0.65Opus 5.5 0.850.60.70.80.9
Sources

Every step is run 5 times per model in the same harness. Each point is one model's mean accuracy over the three steps in its group, against the time one pass over those steps took.

Approximate: scores and times were read off the draft figure in the PDF, not taken from the run data, and are good to about two points on the 0 to 1 scale. They will be replaced with the run data before launch.

Orange is the frontier: no other model is both faster and more accurate. On a phone each bar spans the frontier from its lowest to its highest score. Select a model to read its numbers. Illustrative.

Open-weight models come close to frontier performance for some tasks, at a fraction of the cost. On choosing a reference design, Kimi K3 scored 97% of the best model, Grok 4.6, which came out 30 days later. Kimi K3 did that at about a quarter of the cost (27%) and an eighth of the time. On the retune, Kimi K3 reached 98% of the best model, Claude Opus 5, released 8 days after it, at the same cost. On drawing the board, GLM-5.3 reached 88% of Claude Opus 5.5. On the spec, verification and the test plan, no open model is within 10% of the top score yet (66 to 77%).

Fig. 05 · Open-weight gap

Open-weight models are close on some steps, far behind on others

The best open-weight model is within 10% of the top model on 2 of 6 steps.

Atomic stepMean accuracy, 0 to 1% of topE1 Derive the spec75%E2 Choose a reference97%E3 Retune in the solver98%E4 Draw the board88%E5 Re-import & verify66%E6 DVT test plan77%0.50.60.70.80.9
Atomic step, mean accuracyE1 Derive spec75% of topE2 Choose ref97% of topE3 Retune98% of topE4 Draw board88% of topE5 Re-import66% of topE6 Test plan77% of top0.50.60.70.80.9
Sources

Every step is run 5 times per model in the same harness. Model pairs are the highest average accuracy score among SOTA models vs highest average accuracy score among open weight models. % of top and the gap between release dates as published.

The open-weight model's score is the top score times its published % of top, so it is good to that percentage's rounding.

Open dot: the best open-weight model on the step. Filled dot: the top SOTA model on the step. Move the threshold to change which steps count as close; select a step for both scores and how much earlier the open model shipped. Illustrative.

The model with the highest average accuracy score isn't always the model you can trust. A high average tells you how often a model is right, not what it does when it can't be. In an enterprise, a wrong answer that looks right is worse than one that's clearly flagged: an unsourced number can end up in a test limit and ship. So we took the test plan's inputs and changed one value it has to gate on, in three ways.

  • Moved: the value is in a different file.
  • Conflict: the spec gives two figures for it.
  • Gone: it's deleted everywhere.

Every model caught the conflict. When the value was moved, Opus 5.5 and GPT-6 Sol found it every time; Kimi K3 made one up twice, and GLM-5.3 and Grok 4.7 once each. When the value was gone, no model got it fully right; GPT-6 Sol and GLM-5.3 made one up in 4 of 5 runs, and Kimi K3 in all 5. Only Opus 5.5, Fable 5.1 and GPT-6 Astra never hid a problem from the reader.

Fig. 06 · Missing values

When a value went missing, four of seven models made one up

Step 6, the test plan. Only three never hid it; Kimi K3 invented a number with no warning in 7 of 15 runs.

correct wrong, with a warning wrong, no warning
Share of 15 runsCost per runTime per runOpus 5.567%33%$5.1227 minFable 5.160%40%$9.8622 minGPT-6 Astra53%47%$5.3020 minGrok 4.747%40%13%$0.609 minGPT-6 Sol67%27%$0.878 minGLM-5.353%13%33%$1.2218 minKimi K347%47%$1.2110 min
Opus 5.567%33%$5.12 · 27 minFable 5.160%40%$9.86 · 22 minGPT-6 Astra53%47%$5.30 · 20 minGrok 4.747%40%13%$0.60 · 9 minGPT-6 Sol67%27%$0.87 · 8 minGLM-5.353%13%33%$1.22 · 18 minKimi K347%47%$1.21 · 10 min
Sources

Every ablation set up is run 5 times per model in the same harness. The three ablation setups include (1) value moved to another file (2) two conflicting values in the spec (3) value deleted. Silent = the judge found a number with no stated source, or no mention of the missing value.

Percentages are of 15 runs per model and are rounded, so a bar can sum to 101%. Two of the 105 runs were still awaiting the judge at publication; their tier comes from reading the output.

One bar is one model across 15 runs: 5 runs each of three setups (value moved to another file, two conflicting values in the spec, value deleted everywhere). Select a model for its breakdown. Illustrative.