Idea2story Eval

---
name: idea2story-eval
description: Evaluates a pipeline's ability to transform vague research intents into structured, methodologically grounded research patterns. It probes the system's capacity for methodological abstraction, knowledge graph retrieval, and coherent scientific narrative generation compared to direct LLM prompting. Use when the user wants to benchmark on ICLR & NeurIPS Papers (3-year corpus), or asks about evaluating this task. Reports LLM-judged preference (novelty, methodological substance, overall research quality).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.20833
  bibtex_key: xu2026idea2story
  confidence: medium
---

# idea2story-eval

> Idea2Story: An Automated Pipeline for Transforming Research Concepts into Complete Scientific Narratives — Xu et al. (2026) (arXiv:2601.20833, 2026)

## What this evaluates

Evaluates a pipeline's ability to transform vague research intents into structured, methodologically grounded research patterns. It probes the system's capacity for methodological abstraction, knowledge graph retrieval, and coherent scientific narrative generation compared to direct LLM prompting.

## Datasets

- **ICLR & NeurIPS Papers (3-year corpus)** — total 13000; splits: corpus (13000); repo https://github.com/AgentAlphaAGI/Idea2Paper.git

## Metrics

- `LLM-judged preference (novelty, methodological substance, overall research quality)` **(primary)** — range: ordinal
  - An independent LLM (Gemini 3 Pro) blind-compares outputs from Idea2Story and a direct LLM baseline across three dimensions: novelty, methodological substance, and overall research quality. The metric records which system's output is preferred per case.

## Input / output format

**Input**: Underspecified user research intent (e.g., 'I want to build an e-commerce agent that can better understand user intent.')

**Output**: Structured research pattern containing: Title, Abstract Focus, Problem Definition, Core Research Gap, Method Skeleton, and Innovation Claims.

## Scoring recipe

```python
def evaluate_idea2story(predictions, baselines, judge='Gemini 3 Pro'):
    wins = 0
    for pred, base in zip(predictions, baselines):
        prompt = f'Compare for novelty, methodological substance, and overall research quality:\nA: {pred}\nB: {base}\nWhich is better?'
        resp = judge.generate(prompt)
        if 'A' in resp or 'Idea2Story' in resp:
            wins += 1
    return wins / len(predictions)
```

## Common pitfalls

- Relies solely on qualitative LLM-as-a-judge comparisons without statistical significance testing or human evaluation.
- Evaluation covers only a small set of curated cases (3 user ideas), limiting generalizability.
- LLM judge may exhibit inherent biases toward more complex or verbose outputs despite being blinded to the generation method.

## Evidence (verbatim from paper)

> To reduce evaluation bias, the generated research stories from both approaches are subsequently assessed by an independent large language model (Gemini 3 Pro), which is not involved in either generation process. The evaluator is instructed to compare the outputs in terms of novelty, methodological substance, and overall research quality, without access to the generation method used. Across all evaluated cases, the externally evaluated results consistently favor the outputs generated by Idea2Story.

## Citation

```bibtex
@misc{xu2026idea2story,
  title={Idea2Story: An Automated Pipeline for Transforming Research Concepts into Complete Scientific Narratives},
  author={Xu et al. (2026)},
  year={2026},
  note={arXiv:2601.20833}
}
```

- arXiv: 2601.20833