Field Notes
A collection of research, projects, and teams we’ve learned from and been inspired by
- BenchmarkHWE BenchA benchmark for LLMs on RISC-V CPU design, scored on a real FPGA.hwebench.com (opens in a new tab)
- BenchmarkEEBenchA benchmark for LLMs on electrical engineering, by atopile.eebench.org (opens in a new tab)
- DatasetEMValA benchmark dataset for electromagnetic models, by Arena Physica.huggingface.co (opens in a new tab)
- PostHands-on with Raspberry Pi's CM5 Programming JigJeff Geerling on Raspberry Pi's jig for flashing Compute Module 5 boards.jeffgeerling.com (opens in a new tab)
- BenchmarkBigLaw Bench: ResearchA benchmark for AI on legal research tasks, by Harvey.harvey.ai (opens in a new tab)
- PostDemystifying evals for AI agentsAnthropic on how to design and grade evals for AI agents.anthropic.com (opens in a new tab)
- ProjectReachy MiniAn open-source desktop robot, by Hugging Face and Pollen Robotics.huggingface.co (opens in a new tab)
- PostContext RotChroma on how LLM performance changes as input length grows.trychroma.com (opens in a new tab)
- BenchmarkTerminal-BenchA benchmark for AI agents doing tasks in the terminal.tbench.ai (opens in a new tab)
- ProjectMuJoCo PlaygroundAn open-source library for robot learning in simulation, by Google DeepMind.github.com (opens in a new tab)
- Research paperAdding Error Bars to EvalsEvan Miller (Anthropic), 2024, on statistics for LLM evals.arxiv.org (opens in a new tab)
- Research paperLarge Language MonkeysBrown et al., 2024, on how LLM results change with repeated sampling.arxiv.org (opens in a new tab)
- Benchmarkτ-benchA benchmark for AI agents that talk to users and call tools, by Sierra.taubench.com (opens in a new tab)
- ProjectLeRobotHugging Face's open-source library for real-world robotics.huggingface.co (opens in a new tab)
- BenchmarkSWE-benchA benchmark of real GitHub issues for AI coding agents.swebench.com (opens in a new tab)
- DatasetOpen X-EmbodimentAn open robot-learning dataset from 22 robots across 21 institutions, by the Open X-Embodiment Collaboration.robotics-transformer-x.github.io (opens in a new tab)
- ProjectHELMStanford CRFM's framework for evaluating language models.crfm.stanford.edu (opens in a new tab)
- Research paperDeep Reinforcement Learning at the Edge of the Statistical PrecipiceAgarwal et al., 2021, on reporting results from a few runs with interval estimates.arxiv.org (opens in a new tab)
- ProjectrliableA library for evaluating results from a few runs, by Google Research.github.com (opens in a new tab)
- Research paperBootstrap Methods: Another Look at the JackknifeEfron, 1979: the paper that introduced the bootstrap.projecteuclid.org (opens in a new tab)