Bagel Eval

---
name: bagel-eval
description: BAGEL probes language models' specialized knowledge of animal natural history, including taxonomy, morphology, behavior, habitat, vocalization, and ecological interactions. It evaluates closed-book fact recall and reasoning across diverse source domains (encyclopedic, scientific literature, ecological databases, and bioacoustics) without providing source passages at inference time. Use when the user wants to benchmark on BAGEL, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.16241
  bibtex_key: shen2026bagel
  confidence: high
---

# bagel-eval

> BAGEL: Benchmarking Animal Knowledge Expertise in Language Models — Shen et al. (2026) (arXiv:2604.16241, 2026)

## What this evaluates

BAGEL probes language models' specialized knowledge of animal natural history, including taxonomy, morphology, behavior, habitat, vocalization, and ecological interactions. It evaluates closed-book fact recall and reasoning across diverse source domains (encyclopedic, scientific literature, ecological databases, and bioacoustics) without providing source passages at inference time.

## Datasets

- **BAGEL** — total 11852; splits: test (11852)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard multiple-choice accuracy: the fraction of questions where the model's predicted option exactly matches the single correct answer.

## Input / output format

**Input**: A unified prompt containing only the task instruction, the question stem, and four enumerated answer options.

**Output**: A single selected answer option from the four enumerated choices.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred == gold:
        correct += 1
accuracy = correct / len(predictions)
```

## Common pitfalls

- Models are evaluated with deterministic decoding (seed 0, greedy), so results represent a single run rather than an average across seeds, which may introduce variance.
- Difficulty labels (easy/medium/hard) are reference strata based on agreement between GPT-5.4 and Claude Opus 4.6, not absolute difficulty, and hard items may contain option-level ambiguity.
- The benchmark is strictly closed-book; source passages used during construction are withheld at inference, preventing retrieval-based shortcuts.

## Evidence (verbatim from paper)

> We report accuracy on each source domain and an overall score across domains. All models are evaluated in a closed-book multiple-choice setting. At test time, the model is given a unified prompt containing only the task instruction, the question stem, and four enumerated answer options; the source passage used during benchmark construction is not provided at inference time. We use deterministic decoding with seed 0 and greedy generation for the reported results, so each model is represented by a single run rather than an average across seeds.

## Citation

```bibtex
@misc{shen2026bagel,
  title={BAGEL: Benchmarking Animal Knowledge Expertise in Language Models},
  author={Shen et al. (2026)},
  year={2026},
  note={arXiv:2604.16241}
}
```

- arXiv: 2604.16241