Budget Ai Researcher Eval

---
name: budget-ai-researcher-eval
description: Evaluates a retrieval-augmented generation framework's ability to synthesize novel, feasible, and interesting research abstracts by combining distant topics from AI conference literature. It probes long-range concept recombination and grounded ideation capabilities. Use when the user wants to benchmark on AI Conference Papers (ICLR, NeurIPS, ICML, ACL, ECCV), or asks about evaluating this task. Reports Novelty.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.12317
  bibtex_key: lee2025budgetai
  confidence: medium
---

# budget-ai-researcher-eval

> The Budget AI Researcher and the Power of RAG Chains — Franklin Lee et al. (2025) (arXiv:2506.12317, 2025)

## What this evaluates

Evaluates a retrieval-augmented generation framework's ability to synthesize novel, feasible, and interesting research abstracts by combining distant topics from AI conference literature. It probes long-range concept recombination and grounded ideation capabilities.

## Datasets

- **AI Conference Papers (ICLR, NeurIPS, ICML, ACL, ECCV)** — total 1000; splits: test (1000); repo https://github.com/hellojoeAoPS11235/ai-research-agent

## Metrics

- `Interestingness` — range: 1-10
  - Average rating on a 1-10 scale assigned by a ChatGPT-4o judge using few-shot examples, measuring how engaging the generated abstract is.
- `Novelty` **(primary)** — range: 1-10
  - Average rating on a 1-10 scale assigned by a ChatGPT-4o judge, measuring the originality and newness of the synthesized research idea.
- `Feasibility` — range: 1-10
  - Average rating on a 1-10 scale assigned by a ChatGPT-4o judge, measuring the practicality and realizability of the proposed research.
- `Similarity to Future Publications` — range: [0, 1]
  - Average similarity score from 0 to 1 assigned by Llama 3.1 8B Instant, comparing generated abstracts against 1000 papers from 2024 AI conferences to measure alignment with emerging trends.

## Input / output format

**Input**: Research paper abstracts and texts from selected AI conferences, hierarchical topic lists, and prompts instructing the model to generate a new research abstract combining distant topics.

**Output**: A structured research paper abstract containing a title and an abstract body, optionally followed by an experimental procedure.

## Scoring recipe

```python
def score_abstracts(predictions, gold=None):
    scores = []
    for pred in predictions:
        # LLM judge rates 1-10 based on prompt + reference examples
        rating = call_llm_judge(pred, task="novelty")
        scores.append(rating)
    return sum(scores) / len(scores)

def score_similarity(predictions, gold):
    sim_scores = []
    for pred in predictions:
        sim = call_llm_similarity(pred, context_papers=gold)
        sim_scores.append(sim)
    return sum(sim_scores) / len(sim_scores)
```

## Common pitfalls

- LLM judges may exhibit bias toward familiar baseline models or specific prompt phrasings, inflating or deflating scores.
- High novelty scores often correlate with lower feasibility scores due to the inherent trade-off in recombining distant concepts.
- Knowledge cutoffs differ across baseline models, making direct novelty comparisons sensitive to the temporal distribution of training data.

## Evidence (verbatim from paper)

> For quantitative evaluation, we evaluate the generated ideas from three perspectives (interestingness, novelty, and feasibility) using one or few-shot learning through ChatGPT-4o using examples and ratings from reference (Lu et al. [2024]). Table[1] shows the average rating of sample abstracts generated from each framework.

## Citation

```bibtex
@misc{lee2025budgetai,
  title={The Budget AI Researcher and the Power of RAG Chains},
  author={Franklin Lee et al. (2025)},
  year={2025},
  note={arXiv:2506.12317}
}
```

- arXiv: 2506.12317