Sciqag 24d Eval

---
name: sciqag-24d-eval
description: Evaluates open-ended, closed-book scientific question answering capabilities. It probes a model's ability to generate comprehensive, accurate, and reasonable answers to research-level science questions without external context or reference papers. Use when the user wants to benchmark on SciQAG-24D, SciQ, or asks about evaluating this task. Reports CAR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.09939
  bibtex_key: wan2024sciqag
  confidence: high
---

# sciqag-24d-eval

> SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation — Wan et al. (2024) (arXiv:2405.09939, 2024)

## What this evaluates

Evaluates open-ended, closed-book scientific question answering capabilities. It probes a model's ability to generate comprehensive, accurate, and reasonable answers to research-level science questions without external context or reference papers.

## Datasets

- **SciQAG-24D** — total 188042; splits: train (-1), test (-1)
- **SciQ** — total 11000; splits: train (10000), test (1000)

## Metrics

- `CAR` **(primary)** — range: [0, 5]
  - A modified LLM-judged metric derived from RACAR, explicitly excluding Relevance and Agnosticism. It scores generated answers on three dimensions: Completeness, Accuracy, and Reasonableness, typically on a 5-point scale.
- `accuracy` — range: [0, 1]
  - Standard exact-match accuracy for multiple-choice questions.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall for classification tasks.
- `MAE` — range: other
  - Mean Absolute Error for regression tasks.
- `KL divergence` — range: [0, inf)
  - Kullback-Leibler divergence for transformation tasks.

## Input / output format

**Input**: Open-ended science questions generated from scientific literature, provided without context or reference papers.

**Output**: Open-ended textual answers to the science questions.

## Scoring recipe

```python
def score_car(question, answer):
    prompt = f'Question: {question}\nAnswer: {answer}\nScore Completeness, Accuracy, Reasonableness on 1-5 scale.'
    llm_response = call_llm(prompt)
    scores = extract_scores(llm_response)
    return sum(scores) / len(scores)

def score_accuracy(prediction, gold):
    return 1.0 if prediction.strip().lower() == gold.strip().lower() else 0.0

def score_f1(prediction, gold):
    pred_set = set(prediction.lower().split())
    gold_set = set(gold.lower().split())
    if not pred_set and not gold_set: return 1.0
    prec = len(pred_set & gold_set) / len(pred_set) if pred_set else 0
    rec = len(pred_set & gold_set) / len(gold_set) if gold_set else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- CAR is a modified version of RACAR that explicitly excludes 'Relevance' and 'Agnosticism'; using the full RACAR framework will yield incorrect scores.
- The CAR metric relies on LLM-as-a-judge evaluation, so results are highly sensitive to the judge model, prompt phrasing, and temperature settings.
- SciQAG-24D questions are auto-generated and open-ended; exact string matching is inappropriate, requiring semantic or LLM-based scoring.

## Evidence (verbatim from paper)

> For the evaluation of the quality of generated answers in the zero-shot experiment, we adopt a modified version of the RACAR framework. We exclude the criteria of "Relevance" and "Agnosticism", which are specifically designed to assess the alignment of question-answer (QA) pairs with a given paper, and name this modified metric as CAR (see [Appendix J]).

## Citation

```bibtex
@misc{wan2024sciqag,
  title={SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation},
  author={Wan et al. (2024)},
  year={2024},
  note={arXiv:2405.09939}
}
```

- arXiv: 2405.09939