Tpscalcbench Eval

---
name: tpscalcbench-eval
description: Evaluates large language models' ability to perform analytical calculations in hypersonic thermal protection system engineering using closed-form formulas and thermodynamic relations, without relying on external simulation tools. Use when the user wants to benchmark on TPS-CalcBench, or asks about evaluating this task. Reports relative_error.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.17966
  bibtex_key: zheng2026tpscalcbench
  confidence: high
---

# tpscalcbench-eval

> TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering — Zheng et al. (2026) (arXiv:2604.17966, 2026)

## What this evaluates

Evaluates large language models' ability to perform analytical calculations in hypersonic thermal protection system engineering using closed-form formulas and thermodynamic relations, without relying on external simulation tools.

## Datasets

- **TPS-CalcBench** — total 420; splits: train (-1), test (-1)

## Metrics

- `relative_error` **(primary)** — range: [0, 1]
  - Calculated as |prediction - ground_truth| / ground_truth for each target quantity. Measures outcome correctness.
- `unit_verification` — range: [0, 1]
  - Binary check verifying whether the model's predicted answer matches the expected physical unit specified in the target metadata.
- `process_trustworthiness` — range: [0, 1]
  - Scored via an 8-dimension rubric evaluated by calibrated LLM judges to assess reasoning steps and detect 'right answer, wrong reasoning' failures.

## Input / output format

**Input**: Structured JSON containing problem statement (question), given parameters (name, value, unit, description), target quantities (name, expected unit, scoring weight, description), difficulty level, task type, and domain tags.

**Output**: Numerical result for the target quantity, optionally accompanied by derivation steps or reasoning trace for process evaluation.

## Scoring recipe

```python
def score(prediction, gold, metadata):
    pred_val = extract_numeric(prediction)
    gold_val = gold['value']
    rel_err = abs(pred_val - gold_val) / gold_val
    unit_ok = 1.0 if prediction.unit == metadata['expected_unit'] else 0.0
    process_score = llm_judge_rubric(prediction.reasoning_trace, metadata)
    return rel_err, unit_ok, process_score
```

## Common pitfalls

- Data leakage from textbook chapter patterns if not strictly source-grouped stratified.
- Including simulation-dependent problems that violate the analytical scope boundary.
- Failing to detect 'right answer, wrong reasoning' without process-level rubric scoring.

## Evidence (verbatim from paper)

> It establishes a dual-axis evaluation framework—outcome correctness (via relative error and unit verification) and process trustworthiness (via an 8-dimension rubric scored by calibrated LLM judges)—to detect 'right answer, wrong reasoning' failures.

## Citation

```bibtex
@misc{zheng2026tpscalcbench,
  title={TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering},
  author={Zheng et al. (2026)},
  year={2026},
  note={arXiv:2604.17966}
}
```

- arXiv: 2604.17966