Reasoning Accuracy Eval

---
name: reasoning-accuracy-eval
description: Evaluates large language models' logical reasoning and problem-solving capabilities across mathematical, algorithmic, and creative tasks. It measures both the correctness of final answers and the computational efficiency of the reasoning process. Use when the user wants to benchmark on Game of 24, BIG-Bench (subset), Python Puzzles, MGSM, Shakespearean Sonnet Writing, or asks about evaluating this task. Reports Acc_logic.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.12323
  bibtex_key: yuan2024reversal
  confidence: high
---

# reasoning-accuracy-eval

> Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up — Yuan et al. (2024) (arXiv:2410.12323, 2024)

## What this evaluates

Evaluates large language models' logical reasoning and problem-solving capabilities across mathematical, algorithmic, and creative tasks. It measures both the correctness of final answers and the computational efficiency of the reasoning process.

## Datasets

- **Game of 24** — total 1000; splits: test (1000); repo https://www.4nums.com/game/
- **BIG-Bench (subset)** — total ?; splits: test (-1); HF `google/bigbench`
- **Python Puzzles** — total ?; splits: test (-1); repo https://github.com/microsoft/PythonProgrammingPuzzles
- **MGSM** — total ?; splits: test (-1); repo https://github.com/google-research/url-nlp/tree/main/mgsm
- **Shakespearean Sonnet Writing** — total ?; splits: test (-1); HF `turingmachine/meta-prompting`; repo https://github.com/iljones00/Shakespearean-Sonnets-GPT

## Metrics

- `Acc_logic` **(primary)** — range: percent
  - Percentage of correct responses. An LLM judge (openai-o1) validates the model's final logical reasoning against gold results, returning Correct or Wrong. Accuracy is computed by tallying correct responses over the total number of samples.
- `T` — range: seconds
  - Average inference time per sample. Calculated as T = (1/N) * sum(T_ij) across all samples, where T_ij is the time taken for the i-th task on the j-th sample.

## Input / output format

**Input**: Task prompt with 1-shot or 2-shot few-shot examples provided via OpenAI API.

**Output**: Final logical reasoning answer or solution.

## Scoring recipe

```python
# Accuracy
acc = 0
for pred, gold in zip(predictions, golds):
    if llm_judge(pred, gold) == "Correct":
        acc += 1
acc_logic = (acc / len(predictions)) * 100

# Efficiency
total_time = sum(times)
t_efficiency = total_time / len(predictions)
```

## Common pitfalls

- Accuracy relies on an LLM-as-a-judge (openai-o1), which may introduce bias or inconsistency compared to exact-match or rule-based grading.
- Efficiency metric includes API network latency and server processing time, not just pure model inference time, making cross-environment comparisons difficult.
- Few-shot settings (1-shot vs 2-shot) drastically change performance, so results must always be reported with the specific setting.

## Evidence (verbatim from paper)

> Following Suzgun and Kalai ([2024]); Yao et al. ([2024]), we introduce a LLM to validate the final logical reasoning against gold results (Correct/Wrong). We then compute logical accuracy Acc_logic for each logical task by tallying the number of correct responses. Following Yang et al. ([2024a]), we evaluate reasoning efficiency in terms of complexity by calculating T, calculated as the average time spent per task across all samples in the test dataset:

## Citation

```bibtex
@misc{yuan2024reversal,
  title={Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up},
  author={Yuan et al. (2024)},
  year={2024},
  note={arXiv:2410.12323}
}
```

- arXiv: 2410.12323