Inspiration Retrieval Eval

---
name: inspiration-retrieval-eval
description: Evaluates an LLM's ability to retrieve relevant prior research papers (inspirations) that can inform a given research question from a candidate pool. It measures how well models can surface novel, non-obvious knowledge links through iterative group-based selection. Use when the user wants to benchmark on ResearchBench Inspiration Retrieval, or asks about evaluating this task. Reports Hit Ratio.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.21248
  bibtex_key: [REDACTED]
  confidence: high
---

# inspiration-retrieval-eval

> ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition — Liu et al. (2025) (arXiv:2503.21248, 2025)

## What this evaluates

Evaluates an LLM's ability to retrieve relevant prior research papers (inspirations) that can inform a given research question from a candidate pool. It measures how well models can surface novel, non-obvious knowledge links through iterative group-based selection.

## Datasets

- **ResearchBench Inspiration Retrieval** — total 1386; splits: test (1386)

## Metrics

- `Hit Ratio` **(primary)** — range: percent
  - Number of groundtruth inspiration papers selected by the LLM divided by the total number of inspiration candidates. Evaluated at top 20% (15/75) and top 4% (3/75) retention rates after iterative group selection.

## Input / output format

**Input**: Research question, groundtruth inspirations, negative inspirations, and an inspiration candidate set of 75 papers (each represented by title and abstract). The set is iteratively split into groups of 15, and the LLM selects top 3 per group across rounds.

**Output**: A list of selected paper IDs from the candidate set.

## Scoring recipe

```python
def hit_ratio(predictions, gold):
    hits = sum(1 for p in predictions if p in gold)
    return hits / len(gold)
```

## Common pitfalls

- Iterative group-based selection process differs from standard top-k retrieval, requiring multiple rounds of filtering.
- Position bias and training data co-occurrence can inflate retrieval of statistically closer papers, skewing results.

## Evidence (verbatim from paper)

> We use Hit Ratio as the evaluation metric, it is calculated as the number of groundtruth inspiration papers selected by the LLM divided by the total number of inspiration candidates.

## Citation

```bibtex
@misc{liu2025researchbench,
  title={ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2503.21248}
}
```

- arXiv: 2503.21248