Hypothesis Composition Eval

---
name: hypothesis-composition-eval
description: Assesses an LLM's capability to synthesize novel research hypotheses by combining a given research background with retrieved inspiration papers. It probes the model's ability to mutate and recombine scientific concepts into coherent, groundtruth-aligned proposals. Use when the user wants to benchmark on ResearchBench Hypothesis Composition, or asks about evaluating this task. Reports Normalized Composition Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.21248
  bibtex_key: liu2025researchbench
  confidence: high
---

# hypothesis-composition-eval

> ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition — Liu et al. (2025) (arXiv:2503.21248, 2025)

## What this evaluates

Assesses an LLM's capability to synthesize novel research hypotheses by combining a given research background with retrieved inspiration papers. It probes the model's ability to mutate and recombine scientific concepts into coherent, groundtruth-aligned proposals.

## Datasets

- **ResearchBench Hypothesis Composition** — total 1386; splits: test (1386)

## Metrics

- `Normalized Composition Score` **(primary)** — range: [0, 1]
  - Average score on a 0-5 Likert scale measuring coverage of key points in the groundtruth hypothesis, normalized by dividing by the maximum possible score (5).

## Input / output format

**Input**: Research background, groundtruth inspiration papers (2-3), and prompts for mutate/refine/recombine steps.

**Output**: A composed research hypothesis text.

## Scoring recipe

```python
def normalized_score(predictions, gold):
    raw = llm_judge_score(predictions, gold) # 0 to 5
    return raw / 5.0
```

## Common pitfalls

- Scoring relies on a 6-point Likert scale judged by LLM/experts, which may vary across runs or judges.
- Task uses groundtruth inspirations for evaluation, not retrieved ones, isolating composition ability from retrieval errors.

## Evidence (verbatim from paper)

> we use a 6-point Likert scale (from 0 to 5) to measure whether the generated hypothesis has covered the key points in the groundtruth hypothesis. To compute the generation accuracy, we normalize the average score by dividing it by the maximum possible score (5).

## Citation

```bibtex
@misc{liu2025researchbench,
  title={ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2503.21248}
}
```

- arXiv: 2503.21248