Scholarqa Bench Eval

---
name: scholarqa-bench-eval
description: Evaluates LLMs' ability to synthesize scientific literature by answering open-ended, multi-domain questions using retrieved papers. It probes long-form generation, factual correctness, citation accuracy, and content quality/organization across single- and multi-paper retrieval setups. Use when the user wants to benchmark on ScholarQABench, or asks about evaluating this task. Reports Citation F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.14199
  bibtex_key: asai2024openscholar
  confidence: high
---

# scholarqa-bench-eval

> OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs — Asai et al. (2024) (arXiv:2411.14199, 2024)

## What this evaluates

Evaluates LLMs' ability to synthesize scientific literature by answering open-ended, multi-domain questions using retrieved papers. It probes long-form generation, factual correctness, citation accuracy, and content quality/organization across single- and multi-paper retrieval setups.

## Datasets

- **ScholarQABench** — total 5393; splits: test (5393)

## Metrics

- `Corr` — range: [0, 1]
  - Accuracy for binary tasks (SciFact, PubMedQA), ROUGE-L for QASA, and a weighted LLM score (GPT-4o-turbo) based on expert-annotated rubrics (40% general, 60% annotation-driven) for ScholarQA-CS.
- `Citation F1` **(primary)** — range: [0, 1]
  - Harmonic mean of Citation Precision (relevance and necessity of citations) and Citation Recall (coverage of citation-worthy statements). Evaluated via LLM verification of whether citations support statements and if their removal impacts integrity.
- `LLM-Score` — range: [1, 5]
  - Prometheus v2 assigns 1-5 scale scores for Relevance, Coverage, and Organization based on defined rubrics. Human evaluation also measures Overall Usefulness.

## Input / output format

**Input**: Open-ended scientific question requiring multi-paper retrieval, accompanied by retrieved passages/papers from a large open-access datastore.

**Output**: Long-form natural language response with inline reference numbers (e.g., [1], [2]) linked to the provided passages.

## Scoring recipe

```python
def score_correctness(pred, gold, task):
    if task in ['SciFact', 'PubMedQA']:
        return 1.0 if pred.strip().lower() == gold.strip().lower() else 0.0
    if task == 'QASA':
        return rouge_l_score(pred, gold)
    if task == 'ScholarQA-CS':
        rubric = load_rubric(gold)
        return gpt4o_score(pred, rubric)
    return 0.0

def score_citation_f1(pred):
    statements = extract_statements(pred)
    citations = extract_citations(pred)
    recall = sum(1 for s in statements if has_appropriate_citation(s, citations)) / len(statements)
    precision = sum(1 for c in citations if is_relevant_and_necessary(c, pred)) / len(citations)
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
```

## Common pitfalls

- LLM-based evaluation of long-form answers can be biased by response length or formatting rather than factual accuracy.
- Citation precision/recall relies on LLMs to judge 'necessity' and 'support', which may over-penalize or under-penalize implicit citations.
- Rubric-based scoring for ScholarQA-CS is subjective; annotator agreement varies significantly (59.5–79.3 Pearson correlation).

## Evidence (verbatim from paper)

> For each citation, we then verify its relevance and necessity—specifically, whether the citation supports the statement and if its removal impacts the integrity of remaining citations (Citation Precision, -p). Finally, we compute Citation F1 ( -F1) as well, and use it as a primarily metric for citation accuracy.

## Citation

```bibtex
@misc{asai2024openscholar,
  title={OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs},
  author={Asai et al. (2024)},
  year={2024},
  note={arXiv:2411.14199}
}
```

- arXiv: 2411.14199