EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning

Abstract

Large Language Models are increasingly entering specialized, safety-critical engineering workflows governed by strict quantitative standards and immutable physical laws, making rigorous evaluation of their reasoning capabilities imperative. Existing benchmarks fail to capture the physically grounded reasoning central to engineering, where scientific principles, quantitative modeling, and practical constraints must converge. To enable verifiable process supervision in engineering, we introduce EngTrace, a symbolic benchmark built on 90 parameterized templates generating 1,350 unique, contamination-resistant problem instances, spanning three major engineering branches, nine core domains, and 20 distinct areas. Moving beyond outcome matching, we introduce a verifiable two-stage evaluation framework that validates intermediate reasoning traces alongside final answers through automated procedural checks and a heterogeneous AI Tribunal. Our evaluation of 27 leading LLMs reveals a distinct trade-off between numeric precision and trace fidelity, identifying a complexity cliff where abstract mathematical pre-training fails to translate into the integrative reasoning required for advanced engineering tasks.

Overview and Generation Pipeline

EngTrace is built upon 90 parameterized symbolic templates across 9 core engineering domains and 3 branches, generating 1,350 unique test cases. Difficulty is calibrated along three axes: Conceptual Complexity, Mathematical Sophistication, and Procedural Depth. Domain-aware parameterization ensures physical realism, while symbolic generation makes problems resistant to data contamination.

Every template was validated through a two-stage quality assurance process. An AI Tribunal of three frontier models (GPT-5, Claude Opus 4.5, Gemini 3) first screened each template against a strict multi-axis rubric using median score, majority-vote, and disagreement thresholds. Templates passing this stage then underwent double-blind expert certification, establishing EngTrace as a high-fidelity gold-standard benchmark.

Example of an ENGCHAIN symbolic template for a CSTR Volume Calculation.

Figure 2: EngTrace example template (CSTR Volume Calculation).

A Two-Stage Verifiable Evaluation Framework

We move beyond final-answer accuracy with a tiered verification protocol. Tier 1 performs automated symbolic verification: a reasoning step is valid only if it satisfies both numerical correctness (≤2% relative error) and semantic similarity (Cross-Encoder threshold ≥0.7). If ≥80% of steps pass Tier 1 and the final answer is correct, the chain is accepted immediately.

Chains falling below this threshold are escalated to Tier 2, where the same heterogeneous AI Tribunal assesses whether discrepancies represent valid alternative derivations, calculation errors, or conceptual errors. This yields a Recovered Reasoning F1 score (F1_rec), computed via Hungarian optimal step-alignment, that fairly credits alternative valid solution paths without penalizing sound but non-canonical reasoning.

Results

Across 27 LLMs spanning frontier, open-weights, and math-enhanced categories, frontier models lead overall: Gemini 3.1 Pro tops Final Answer Accuracy (65.41%) while DeepSeek R1 leads Reasoning F1 (44.41%). Open-weights models lag substantially on both metrics despite competitive semantic scores. Math-enhanced 7B models underperform general-purpose counterparts of equivalent size, confirming that abstract mathematical pre-training does not transfer to physically constrained engineering reasoning.

Overall Model Performance

Table showing zero-shot performance of 27 models on the EngTrace benchmark.

Table 1: Overall Performance of 27 Models on the EngTrace benchmark (N=1350).

Branch-Level Performance

Bar chart showing model performance across Chemical, Electrical, and Mechanical Engineering.

Figure 3: Chemical Engineering is hardest; Mechanical the most accessible — consistent across all model families.

Domain-Level Performance

Radar plot showing spiky performance across 9 engineering domains.

Figure 4: The spiky profile reveals specialization rather than generalized engineering reasoning.

Performance Across Difficulty Levels

Bar chart showing model performance across Easy, Intermediate, and Advanced difficulty levels.

Figure 5: The "complexity cliff" — open-weights models collapse at Advanced tasks while frontier models remain stable.

Error Analysis

Beyond the automated evaluation, human domain experts applied a finer six-category error taxonomy to 2,200 annotated failure traces, stratified by difficulty tier, to diagnose root causes independently of the automated pipeline (inter-annotator agreement: Fleiss' κ = 0.524, Gwet's AC1 = 0.682).

Gemini 3.1 Pro (FAC leader) fails overwhelmingly on arithmetic (79.5% Calculation Errors), consistently identifying the correct governing equation — suggesting arithmetic execution, not physical reasoning, is the primary frontier bottleneck. DeepSeek R1 (F1 leader) shows a more distributed profile: Setup/Assumption errors rise to 21.5% (nearly double Gemini's rate), explaining the FAC-vs-F1 tension — step-level reasoning is sound but systematic problem-framing errors propagate to wrong final answers. Llama 3.1 70B tells a different story: Formula/Principle errors surge from 14.5% at Easy to 49.0% at Advanced, a 41.8 pp conceptual cliff, revealing genuine breakdown in domain knowledge rather than deteriorating arithmetic.

Error category distributions for Gemini 3.1 Pro, DeepSeek R1, and Llama 3.1 70B.

Figure 6: Error distributions for three representative models across the six-category taxonomy.