Engineering benchmark tasks for evaluating AI models

Challenge

The client needed hard, realistic engineering problems with defensible correct answers to evaluate large language models. Answers had to be reproducible and auditable. Problems had to include the traps that catch a pattern-matching solver.

Approach

We wrote structural-thermal-optical tasks spanning athermal lens mounts, mirror gravity release, cryogenic imagers, thermal lensing, and pointing jitter. Each reference answer comes from an open-source closed loop of optical design, parametric CAD, meshing, and finite element analysis. Each task ships with a point-weighted rubric that rewards catching the seeded traps.

Result

Five tasks were delivered and accepted by the client, each with a complete reference solution. Key results were cross-checked against closed-form theory and independent finite element reruns.

Key figures

• 5

  benchmark tasks delivered and accepted

• ≤2 µm

  agreement between closed-loop prediction and independent FEA

• <1%

  finite element result versus closed-form check