Ideation Space Eval

---
name: ideation-space-eval
description: Evaluates a framework's ability to decompose scientific papers into orthogonal conceptual dimensions (problem, method, findings) and model transitions between them. It probes fine-grained conceptual similarity retrieval and assesses whether the model's novelty predictions align with expert human judgments. Use when the user wants to benchmark on ICLR 2025 Submissions, AI-Researcher, or asks about evaluating this task. Reports Recall@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.08901
  bibtex_key: shen2026navigating
  confidence: high
---

# ideation-space-eval

> Navigating Ideation Space: Decomposed Conceptual Representations for Positioning Scientific Ideas — Shen et al. (2026) (arXiv:2601.08901, 2026)

## What this evaluates

Evaluates a framework's ability to decompose scientific papers into orthogonal conceptual dimensions (problem, method, findings) and model transitions between them. It probes fine-grained conceptual similarity retrieval and assesses whether the model's novelty predictions align with expert human judgments.

## Datasets

- **ICLR 2025 Submissions** — total 500; splits: test (466); repo https://github.com/PLUM-Lab/IdeationSpace
- **AI-Researcher** — total 93; splits: test (93)

## Metrics

- `Recall@K` **(primary)** — range: [0, 1]
  - Fraction of ground-truth relevant papers retrieved within the top-K ranked results.
- `NDCG@K` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank K, measuring the quality of the ranked list.
- `Hit Rate@K` — range: [0, 1]
  - Binary indicator: 1 if at least one relevant paper appears in the top-K results, 0 otherwise.
- `Pearson correlation` — range: [-1, 1]
  - Linear correlation coefficient between the model's predicted novelty scores and expert-annotated novelty scores.
- `Spearman correlation` — range: [-1, 1]
  - Rank-based correlation coefficient measuring the monotonic relationship between predicted and expert novelty scores.

## Input / output format

**Input**: For retrieval: query vectors extracted from paper components (research problem, method approach, key findings) or concatenated title+abstract text. For novelty: decomposed paper components and retrieved prior work used to compute a novelty score.

**Output**: For retrieval: a ranked list of candidate papers. For novelty: a scalar novelty score.

## Scoring recipe

```python
def compute_recall_at_k(retrieved_top_k, relevant_papers):
    return len(set(retrieved_top_k) & set(relevant_papers)) / len(relevant_papers)

def compute_ndcg_at_k(retrieved_top_k, relevant_papers):
    dcg = sum(1 / log2(i + 2) for i, p in enumerate(retrieved_top_k) if p in relevant_papers)
    idcg = sum(1 / log2(i + 2) for i in range(min(len(relevant_papers), len(retrieved_top_k))))
    return dcg / idcg if idcg > 0 else 0.0

def compute_hit_rate_at_k(retrieved_top_k, relevant_papers):
    return 1.0 if any(p in relevant_papers for p in retrieved_top_k) else 0.0

def compute_correlation(predicted_scores, expert_scores):
    return pearsonr(predicted_scores, expert_scores)[0]
```

## Common pitfalls

- Table K values are scaled by 3x (displayed 9, 30, 90 correspond to actual K=3, 10, 30).
- Retrieval ground truth is constructed via GPT-5.2 processing references and OpenReview reviews, potentially introducing LLM bias.
- Novelty baselines use varying input configurations (title, abstract, concat), making direct comparison sensitive to input length/format.

## Evidence (verbatim from paper)

> We report Recall@K, NDCG@K, and Hit Rate@K to measure retrieval quality. We measure the correlation between predicted novelty scores and ground-truth expert scores using both Pearson and Spearman correlation coefficients.

## Citation

```bibtex
@misc{shen2026navigating,
  title={Navigating Ideation Space: Decomposed Conceptual Representations for Positioning Scientific Ideas},
  author={Shen et al. (2026)},
  year={2026},
  note={arXiv:2601.08901}
}
```

- arXiv: 2601.08901