Scientific Ideation Eval

---
name: scientific-ideation-eval
description: Evaluates the ability of LLMs to generate novel, feasible, and effective scientific research ideas given a research question. Probes open-ended scientific reasoning and ideation quality under compute-matched inference budgets. Use when the user wants to benchmark on ICLR 2024 & NeurIPS 2025, or asks about evaluating this task. Reports Absolute Novelty.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.16723
  bibtex_key: [REDACTED]
  confidence: high
---

# scientific-ideation-eval

> Debate as Reward: A Multi-Agent Reward System for Scientific Ideation via RL Post-Training — Salimi et al. (2026) (arXiv:2604.16723, 2026)

## What this evaluates

Evaluates the ability of LLMs to generate novel, feasible, and effective scientific research ideas given a research question. Probes open-ended scientific reasoning and ideation quality under compute-matched inference budgets.

## Datasets

- **ICLR 2024 & NeurIPS 2025** — total 80; splits: test (80); repo https://github.com/mnsalimi/Debate-as-Reward-A-Multi-Agent-Reward-System-for-Scientific-Ideation-via-RL-Post-Training

## Metrics

- `Absolute Novelty` **(primary)** — range: [1, 5]
  - LLM judge (Qwen 2.5 72B) independently rates each generated idea on a fixed integer scale from 1 to 5, where 5 indicates the highest quality. Other absolute metrics (Feasibility, Effectiveness) use the same protocol.
- `Pairwise Novelty` — range: [1, 5]
  - LLM judge evaluates every unordered pair of ideas twice (swapped order to mitigate position bias). Preferences are mapped to +1, -1, or 0, aggregated per idea, and linearly normalized to the 1–5 range.

## Input / output format

**Input**: Research question (extracted from paper abstract/full text or provided by PhD candidates)

**Output**: Generated scientific idea/abstract text

## Scoring recipe

```python
def score_ideas(predictions, judge_model):
    # Absolute scoring
    abs_scores = [judge_model.rate_1_to_5(idea) for idea in predictions]
    
    # Pairwise scoring
    pairwise_scores = []
    for i, idea_i in enumerate(predictions):
        votes = 0
        for j, idea_j in enumerate(predictions):
            if i == j: continue
            # Compare in both orders to mitigate position bias
            votes += judge_model.compare(idea_i, idea_j)  # +1/-1/0
            votes += judge_model.compare(idea_j, idea_i)  # +1/-1/0
        # Linearly normalize aggregated votes to [1, 5]
        normalized = linear_normalize(votes, min_possible, max_possible, 1, 5)
        pairwise_scores.append(normalized)
    return abs_scores, pairwise_scores
```

## Common pitfalls

- LLM evaluators may be biased by familiarity with post-cutoff papers; the protocol mitigates this by using an evaluator with a matching knowledge cutoff date.
- Position bias in pairwise comparisons must be explicitly mitigated by evaluating each idea pair in both orders.
- A known trade-off exists between novelty and feasibility, causing LLMs to penalize highly novel ideas as less feasible.

## Evidence (verbatim from paper)

> In the absolute scoring setup, the evaluator assigns each idea an integer score from 1 to 5, with 5 indicating the highest quality. For pairwise comparisons, we present every unordered pair of ideas twice—once in each order—to mitigate position bias. The model’s judgment for each comparison is mapped to a numerical value: +1 if the first idea is preferred, –1 if the second is preferred, and 0 if they are deemed equivalent. We aggregate these values across all comparisons for each idea and linearly normalize the resulting scores to the 1–5 range, where 5 again corresponds to the highest-performing method. Table 2 presents the absolute and pairwise evaluation scores of our model and baselines across three key dimensions: novelty, feasibility, and effectiveness.

## Citation

```bibtex
@misc{salimi2026debateasreward,
  title={Debate as Reward: A Multi-Agent Reward System for Scientific Ideation via RL Post-Training},
  author={Salimi et al. (2026)},
  year={2026},
  note={arXiv:2604.16723}
}
```

- arXiv: 2604.16723