Rulereasoner Eval

---
name: rulereasoner-eval
description: This evaluation probes a language model's ability to perform rule-based logical reasoning on both in-distribution and out-of-distribution tasks. It measures how well the model can apply explicit and implicit logical rules to derive correct answers under strict exact-match conditions. Use when the user wants to benchmark on BigBench Hard (BBH), BigBench Extra Hard (BBEH), ProverQA, or asks about evaluating this task. Reports pass@1 (hard exact match).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.08672
  bibtex_key: liu2025rulereasoner
  confidence: high
---

# rulereasoner-eval

> RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling — Liu et al. (2025) (arXiv:2506.08672, 2025)

## What this evaluates

This evaluation probes a language model's ability to perform rule-based logical reasoning on both in-distribution and out-of-distribution tasks. It measures how well the model can apply explicit and implicit logical rules to derive correct answers under strict exact-match conditions.

## Datasets

- **BigBench Hard (BBH)** — total 750; splits: test (750)
- **BigBench Extra Hard (BBEH)** — total 400; splits: test (400)
- **ProverQA** — total 1500; splits: test (1500)

## Metrics

- `pass@1 (hard exact match)` **(primary)** — range: percent
  - Algebraic mean of hard exact match across all test instances. Equivalent to pass@1 accuracy under strict extraction and comparison.

## Input / output format

**Input**: Logical reasoning prompts containing a set of rules, facts, and a question (e.g., 'If X then Y. Fact: X. Question: Y?'). Includes both explicit rule sets and implicit contextual descriptions.

**Output**: Step-by-step reasoning chain followed by a final answer formatted as 'Answer: [value].' (e.g., 'Answer: True.' or 'Answer: A.')

## Scoring recipe

```python
def compute_pass_at_1(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        # Strict extraction: parse the final answer line
        pred_ans = extract_last_answer(pred)
        if pred_ans == gold:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Using micro-averaged accuracy instead of the specified algebraic (macro) mean across tasks/domains.
- Failing to apply strict extraction for the final answer, which can cause valid reasoning chains to be marked incorrect if the answer format deviates.
- Confusing in-distribution (ID) and out-of-distribution (OOD) evaluation splits, as the paper reports separate results for each.

## Evidence (verbatim from paper)

> Evaluation Metrics. All tasks in the work are evaluated using the algebraic mean of hard exact match, which is also equivalent to pass@1 accuracy under strict extraction and comparison.

## Citation

```bibtex
@misc{liu2025rulereasoner,
  title={RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2506.08672}
}
```

- arXiv: 2506.08672