Scivisagentbench Eval

---
name: scivisagentbench-eval
description: Evaluates agentic systems' ability to perform scientific data analysis and visualization workflows. It probes outcome correctness, process behavior, and computational efficiency across real-world scientific domains and tools. Use when the user wants to benchmark on SciVisAgentBench, or asks about evaluating this task. Reports outcome correctness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.29139
  bibtex_key: ai2026scivisagentbench
  confidence: medium
---

# scivisagentbench-eval

> SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents — Ai et al. (2026) (arXiv:2603.29139, 2026)

## What this evaluates

Evaluates agentic systems' ability to perform scientific data analysis and visualization workflows. It probes outcome correctness, process behavior, and computational efficiency across real-world scientific domains and tools.

## Datasets

- **SciVisAgentBench** — total ?; splits: test (-1)

## Metrics

- `outcome correctness` **(primary)** — range: [0, 1]
  - Assessed via a multimodal outcome-centric evaluation pipeline combining LLM-based judging with deterministic verifiers, including image metrics, code checkers, and rule-based validators.
- `execution cost` — range: other
  - Quantified through time and token usage, as well as benchmark-level computational tractability to ensure repeated evaluations without excessive overhead.

## Input / output format

**Input**: Task specifications grounded in real-world SciVis tools, pipelines, and data formats; agents operate over authentic visualization environments (e.g., ParaView, napari, VMD).

**Output**: Visualization outcomes, states, or scripts that serve as the reference for evaluation.

## Scoring recipe

```python
# Pseudo-code based on the multimodal outcome-centric evaluation pipeline
def evaluate(agent, task):
    outcome = agent.execute(task)
    # Combine LLM judging with deterministic verifiers
    llm_verdict = llm_judge(outcome, task.reference)
    det_verdict = verify(outcome, task.reference, 
                         tools=["image_metrics", "code_checkers", "rule_based_validators"])
    score = combine(llm_verdict, det_verdict)
    cost = measure_time_and_tokens(agent)
    return score, cost
```

## Common pitfalls

- Stochastic model behavior can hinder full determinism, requiring repeated trials and consistency measures to quantify variability.
- Process-level trajectory assessment is currently infeasible due to reproducibility and stability challenges, limiting evaluation to outcome correctness.
- Multimodal reasoning and prompt robustness limitations may cause misalignment between human experts and LLM judges.

## Evidence (verbatim from paper)

> Outcome correctness serves as the primary evaluation axis in the current release, while process-level analysis remains future work due to reproducibility and stability challenges in trajectory assessment. Efficiency is quantified through execution cost, including time and token usage, as well as benchmark-level computational tractability, ensuring that evaluations can be run repeatedly without excessive overhead.

## Citation

```bibtex
@misc{ai2026scivisagentbench,
  title={SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents},
  author={Ai et al. (2026)},
  year={2026},
  note={arXiv:2603.29139}
}
```

- arXiv: 2603.29139