Slidequest Eval

---
name: slidequest-eval
description: Evaluates an agentic framework's ability to translate natural language scientific queries into executable Python pipelines for automated histopathology analysis on whole-slide images, requiring multi-step computational reasoning rather than simple knowledge recall or diagnosis. Use when the user wants to benchmark on SlideQuest, or asks about evaluating this task. Reports task_success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.11324
  bibtex_key: vaidya2025nova
  confidence: medium
---

# slidequest-eval

> NOVA: An Agentic Framework for Automated Histopathology Analysis and Discovery — Vaidya et al. (2025) (arXiv:2511.11324, 2025)

## What this evaluates

Evaluates an agentic framework's ability to translate natural language scientific queries into executable Python pipelines for automated histopathology analysis on whole-slide images, requiring multi-step computational reasoning rather than simple knowledge recall or diagnosis.

## Datasets

- **SlideQuest** — total 90; splits: test (90)

## Metrics

- `task_success_rate` **(primary)** — range: [0, 1]
  - Proportion of the 90 benchmark queries for which the generated Python pipeline executes successfully and correctly answers the query. Not explicitly named in the provided text; inferred from the benchmark's focus on computational problem solving and baseline comparisons.

## Input / output format

**Input**: Natural language scientific queries requiring histopathology analysis on whole-slide images.

**Output**: Executable Python pipelines/code for automated analysis.

## Scoring recipe

```python
correct = 0
for query in SlideQuest:
    code = model.generate(query)
    if execute_and_verify(code, query):
        correct += 1
return correct / len(SlideQuest)
```

## Common pitfalls

- LLM baselines are constrained to the same Python interpreter and library set, so performance differences stem from agentic reasoning and tool use, not environment setup.
- Experiments are repeated 3 times to account for LLM stochasticity, so results should report mean ± variance rather than single runs.
- Max iterations capped at 20 for all baselines, preventing infinite loops but potentially truncating complex multi-step reasoning.

## Evidence (verbatim from paper)

> We benchmark Nova using Azure OpenAI LLM endpoints. All experiments are conducted on machines with a single NVIDIA A100 GPU within Azure ML. To account for [LLM (large language model)] stochasticity and to quantify variability, each experiment is repeated three times. We compare Nova against LLM only, LLM with PI, and LLM with PI and retries, all constrained to the same maximum number of iterations (20).

## Citation

```bibtex
@misc{vaidya2025nova,
  title={NOVA: An Agentic Framework for Automated Histopathology Analysis and Discovery},
  author={Vaidya et al. (2025)},
  year={2025},
  note={arXiv:2511.11324}
}
```

- arXiv: 2511.11324