Lab Bench Eval

---
name: lab-bench-eval
description: This benchmark evaluates large language models' capabilities in biology research, including literature retrieval, figure/table interpretation, database querying, protocol troubleshooting, and DNA/protein sequence manipulation. It probes whether models can perform multi-step, tool-dependent scientific reasoning or rely on memorization and heuristic guesswork. Use when the user wants to benchmark on LAB-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.10362
  bibtex_key: [REDACTED]
  confidence: high
---

# lab-bench-eval

> LAB-Bench: Measuring Capabilities of Language Models for Biology Research — Laurent et al. (2024) (arXiv:2407.10362, 2024)

## What this evaluates

This benchmark evaluates large language models' capabilities in biology research, including literature retrieval, figure/table interpretation, database querying, protocol troubleshooting, and DNA/protein sequence manipulation. It probes whether models can perform multi-step, tool-dependent scientific reasoning or rely on memorization and heuristic guesswork.

## Datasets

- **LAB-Bench** — total 2457; splits: test (2457)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Number of questions answered correctly divided by the total number of questions (correct / all).
- `precision` — range: [0, 1]
  - Number of questions answered correctly divided by the number of questions attempted or covered (correct / attempted).

## Input / output format

**Input**: Multiple-choice questions tailored to specific subtasks. Inputs vary by category: plain text for LitQA2/SuppQA/ProtocolQA, images of figures/tables for FigQA/TableQA, biological sequences and enzyme details for SeqQA/CloningScenarios, and database identifiers for DbQA. Models receive a question, multiple options, and an explicit option to decline answering.

**Output**: Free-text completion containing the selected multiple-choice option letter/text, or a decline statement. Answers are parsed directly from the text output.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    total = len(gold_labels)
    attempted = sum(1 for p in predictions if p is not None and p != 'decline')
    accuracy = correct / total if total > 0 else 0.0
    precision = correct / attempted if attempted > 0 else 0.0
    return {'accuracy': accuracy, 'precision': precision}
```

## Common pitfalls

- Models often achieve high accuracy through distractor elimination heuristics or memorization rather than genuine reasoning or tool use.
- High coverage rates can mask poor reasoning; models may guess frequently, inflating accuracy while lacking reliability.
- Multi-modal subtasks (FigQA, TableQA) require image inputs, but text-only models cannot be evaluated on them, leading to missing data if not handled.

## Evidence (verbatim from paper)

> Therefore, for each question, models were given a specific option to decline to answer for lack of information. This allows us to report performance both as accuracy (number of questions correct / number of questions) and precision (number of questions correct / coverage) where coverage is (number of questions attempted).

## Citation

```bibtex
@misc{laurent2024labbench,
  title={LAB-Bench: Measuring Capabilities of Language Models for Biology Research},
  author={Laurent et al. (2024)},
  year={2024},
  note={arXiv:2407.10362}
}
```

- arXiv: 2407.10362