Guide Research Idea Eval

---
name: guide-research-idea-eval
description: This evaluation probes a system's ability to act as a scientific advisor by predicting whether research hypotheses will be accepted at a top-tier AI conference. It measures alignment with expert peer-review decisions using ranking-based precision and recall metrics on a held-out set of conference submissions. Use when the user wants to benchmark on ICLR 2025 Submissions Test Set, or asks about evaluating this task. Reports Top-30% Precision.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.08870
  bibtex_key: liu2025guide
  confidence: high
---

# guide-research-idea-eval

> GUIDE: Towards Scalable Advising for Research Ideas — Liu et al. (2025) (arXiv:2507.08870, 2025)

## What this evaluates

This evaluation probes a system's ability to act as a scientific advisor by predicting whether research hypotheses will be accepted at a top-tier AI conference. It measures alignment with expert peer-review decisions using ranking-based precision and recall metrics on a held-out set of conference submissions.

## Datasets

- **ICLR 2025 Submissions Test Set** — total 1000; splits: test (1000)

## Metrics

- `Top-5% Precision` — range: percent
  - Among all the hypotheses with the top-5% highest predicted rating, the proportion that were actually accepted.
- `Top-30% Precision` **(primary)** — range: percent
  - Among all the hypotheses with the top-30% highest predicted scores, the proportion that were actually accepted.
- `Accept Recall` — range: percent
  - Among all the hypotheses that were accepted by ICLR 2025, the proportion that appear within the top 30% predictions.

## Input / output format

**Input**: Hypothesis abstract, claimed contribution, method description, experimental setup, and the ten most relevant literature sections retrieved from a database of ICLR papers (2016–2024).

**Output**: A predicted rating distribution over 10 classes (used to rank hypotheses) and optional text-based advice/evaluation.

## Scoring recipe

```python
import numpy as np
def compute_metrics(predictions, gold):
    n = len(predictions)
    ranked_idx = np.argsort(predictions)[::-1]
    top_5 = ranked_idx[:int(n * 0.05)]
    top_30 = ranked_idx[:int(n * 0.30)]
    top_5_prec = np.mean([gold[i] for i in top_5])
    top_30_prec = np.mean([gold[i] for i in top_30])
    accept_rec = np.mean([gold[i] for i in top_30]) / np.sum(gold)
    return top_5_prec, top_30_prec, accept_rec
```

## Common pitfalls

- Confusing Top-k% Precision (fraction of top-ranked items that are accepted) with Accept Recall (fraction of all accepted items that appear in the top-k%).
- The test set consists of pre-publication ICLR 2025 submissions, so results may not generalize to fully published papers or other conferences.
- Uncertainty filtering changes the effective test set size; precision scores reported under confidence thresholds apply only to the filtered subset, not the full 1,000 papers.

## Evidence (verbatim from paper)

> To measure the advising system's alignment with human experts, the following metrics are adopted, 1. Top-5% Precision: Among all the hypotheses with the top-5% highest predicted rating, the proportion that were actually accepted. 2. Top- 30% Precision: Among all the hypotheses with the top- 30% highest predicted scores, the proportion that were actually accepted. 3. Accept Recall: Among all the hypotheses that were accepted by ICLR 2025, the proportion that appear within the top 30% predictions.

## Citation

```bibtex
@misc{liu2025guide,
  title={GUIDE: Towards Scalable Advising for Research Ideas},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2507.08870}
}
```

- arXiv: 2507.08870