Hypothesis Ranking Eval

---
name: hypothesis-ranking-eval
description: Measures an LLM's ability to correctly rank a groundtruth hypothesis against a set of negative hypotheses using pairwise comparisons. It evaluates discriminative judgment in scientific reasoning. Use when the user wants to benchmark on ResearchBench Hypothesis Ranking, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.21248
  bibtex_key: liu2025researchbench
  confidence: high
---

# hypothesis-ranking-eval

> ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition — Liu et al. (2025) (arXiv:2503.21248, 2025)

## What this evaluates

Measures an LLM's ability to correctly rank a groundtruth hypothesis against a set of negative hypotheses using pairwise comparisons. It evaluates discriminative judgment in scientific reasoning.

## Datasets

- **ResearchBench Hypothesis Ranking** — total 1386; splits: test (1386)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Proportion of correct pairwise rankings out of 15 comparisons. Each pair is compared twice with reversed positions to mitigate position bias, and results are averaged.

## Input / output format

**Input**: A set of 16 hypotheses (1 groundtruth, 15 negative) derived from background and inspirations. Presented as pairwise comparisons.

**Output**: A binary preference or ranking decision for each pair.

## Scoring recipe

```python
def ranking_accuracy(predictions, gold):
    correct = sum(1 for p in predictions if p == gold)
    return correct / 15.0
```

## Common pitfalls

- Strong position bias affects many models; requires double-pass evaluation with reversed positions to average out.
- Negative hypotheses are generated via the same framework, not human-written, which may limit diversity.

## Evidence (verbatim from paper)

> Accuracy is used as the evaluation metric, which is calculated as the proportion of correct pairwise rankings out of 15 comparisons. During the pairwise evaluation, we find that many LLMs have strong position bias: they largely prefer the first hypothesis than the second. To avoid this bias, for each hypothesis pair, we compare them twice with reverse positions, and the results are averaged.

## Citation

```bibtex
@misc{liu2025researchbench,
  title={ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2503.21248}
}
```

- arXiv: 2503.21248