Bioagent Bench Eval

---
name: bioagent-bench-eval
description: Evaluates AI agents' ability to execute multi-step bioinformatics pipelines (e.g., RNA-seq, variant calling) under normal and perturbed conditions. It probes step-level reasoning, tool-use robustness, and the capacity to produce correctly formatted final artifacts despite input corruption or prompt bloat. Use when the user wants to benchmark on BioAgent Bench Tasks, or asks about evaluating this task. Reports completion rate (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.21800
  bibtex_key: fa2026bioagentbench
  confidence: high
---

# bioagent-bench-eval

> BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics — Fa et al. (2026) (arXiv:2601.21800, 2026)

## What this evaluates

Evaluates AI agents' ability to execute multi-step bioinformatics pipelines (e.g., RNA-seq, variant calling) under normal and perturbed conditions. It probes step-level reasoning, tool-use robustness, and the capacity to produce correctly formatted final artifacts despite input corruption or prompt bloat.

## Datasets

- **BioAgent Bench Tasks** — total ?; splits: test (-1)

## Metrics

- `completion rate (%)` **(primary)** — range: percent
  - Percentage of required pipeline steps that pass the LLM grader's check for artifact generation and format compliance.
- `f1_score` — range: [0, 1]
  - F1-score computed where applicable (only for the GIAB dataset).
- `final_result_reached` — range: boolean
  - Binary flag indicating whether the agent produced the final requested result artifact.
- `results_match` — range: boolean
  - Task-specific correctness flag derived from rubric rules.

## Input / output format

**Input**: System prompt, input data files, and prompt instructions specifying the goal and expected output format (CSV/TSV).

**Output**: Generated intermediate output files, a final result file (CSV/TSV table as text), and an execution trace (folders and file paths).

## Scoring recipe

```python
def grade_run(input_files, reference_data, expected_outcome, agent_outcome, agent_trace, grading_prompt):
    # LLM Grader (GPT-5.1) evaluates against rubric
    steps_completed = llm_output.steps_completed
    final_result_reached = llm_output.final_result_reached
    results_match = llm_output.results_match
    f1_score = llm_output.f1_score if dataset == 'giab' else None
    return steps_completed, final_result_reached, results_match, f1_score

def completion_rate(task_steps, grading_results):
    passed_steps = sum(1 for step in task_steps 
                       if grading_results[step].final_result_reached 
                       and grading_results[step].results_match)
    return (passed_steps / len(task_steps)) * 100
```

## Common pitfalls

- Multiple valid solution paths and tool choices mean there is no single canonical ground truth, making deterministic scoring impossible.
- Performance is tightly coupled to the specific harness (e.g., Claude Code, Codex CLI) and sandbox configuration, not just the base LLM.
- LLM grading introduces subjectivity and potential bias, especially when evaluating numerical accuracy versus pipeline completion.

## Evidence (verbatim from paper)

> Our primary metric is completion rate (%). For each task, we evaluate whether the agent completes each required pipeline step and produces the requested final artifact in the specified format (CSV/TSV). The completion rate is the percentage of required steps that pass this check, as assessed by the LLM grader.

## Citation

```bibtex
@misc{fa2026bioagentbench,
  title={BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics},
  author={Fa et al. (2026)},
  year={2026},
  note={arXiv:2601.21800}
}
```

- arXiv: 2601.21800