Rubric Reward Eval

---
name: rubric-reward-eval
description: Evaluates whether data-driven reasoning rubrics improve LLM-based trace correctness classification and serve as effective reward signals for reinforcement learning compared to standard LLM judges and verifiable rewards. Use when the user wants to benchmark on SWE-Bench, NuminaMath, NaturalReasoning, or asks about evaluating this task. Reports Balanced Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.06795
  bibtex_key: sanders2026generating
  confidence: high
---

# rubric-reward-eval

> Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling — Sanders et al. (2026) (arXiv:2602.06795, 2026)

## What this evaluates

Evaluates whether data-driven reasoning rubrics improve LLM-based trace correctness classification and serve as effective reward signals for reinforcement learning compared to standard LLM judges and verifiable rewards.

## Datasets

- **SWE-Bench** — total ?; splits: train (1400), test (80); repo https://github.com/SWE-bench/experiments/tree/main/evaluation/verified/20241022_tools_claude-3-5-haiku
- **NuminaMath** — total ?; splits: train (1400), val (340); HF `open-r1/OpenR1-Math-220k`
- **NaturalReasoning** — total ?; splits: train (-1), val (-1); HF `RJT1990/GeneralThoughtArchive`

## Metrics

- `Specificity` — range: [0, 1]
  - TN / (FP + TN). The percentage of incorrect traces correctly classified as incorrect.
- `Balanced Accuracy` **(primary)** — range: [0, 1]
  - (TN / (2 * (FP + TN))) + (TP / (2 * (TN + TP))). The average of specificity and recall.
- `F0.5` — range: [0, 1]
  - (1 + 0.5^2) * (P * R) / (0.5^2 * P + R). Balanced precision and recall with additional weight on precision.
- `Patch Completion Rate` — range: [0, 1]
  - Count of non-empty patches that pass execution tests divided by total non-empty patches.

## Input / output format

**Input**: Problem statement, model-generated reasoning trace, and optionally ground truth answer or domain-specific rubric items.

**Output**: Binary correctness label (0 or 1) for trace classification, or final answer/patch for downstream evaluation.

## Scoring recipe

```python
def compute_metrics(tp, fp, tn, fn):
    specificity = tn / (fp + tn)
    balanced_acc = (tn / (2 * (fp + tn))) + (tp / (2 * (tn + tp)))
    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    f05 = (1 + 0.5**2) * (precision * recall) / (0.5**2 * precision + recall)
    return specificity, balanced_acc, f05
```

## Common pitfalls

- Long reasoning traces are truncated to fit model context windows, which may alter correctness classification.
- Intermediate reasoning errors do not always lead to incorrect final answers, creating a tradeoff between specificity and recall.
- SWE-Bench unit tests are computationally expensive, preventing full benchmark evaluation during RL training loops.

## Evidence (verbatim from paper)

> For experiments assessing rubric quality as an artifact for trace classification, we consider the following metrics: Specificity, or $\frac{TN}{FP+TN}$ (the percentage of incorrect traces classified as incorrect by the LLM), balanced accuracy, or $\frac{TN}{2(FP+TN)}+\frac{TP}{2(TN+TP)}$ (the average of the specificity and recall), and F0.5, $(1+\beta^{2})\frac{PR}{\beta^{2}P+R}$ where $\beta\=0.5$, $P\=\frac{TP}{TP+FP}$, and $R$ is recall.

## Citation

```bibtex
@misc{sanders2026generating,
  title={Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling},
  author={Sanders et al. (2026)},
  year={2026},
  note={arXiv:2602.06795}
}
```

- arXiv: 2602.06795