Sparta Eval

---
name: sparta-eval
description: Evaluates the ability of LLMs and hybrid QA systems to perform tree-structured, multi-hop reasoning over combined text and table data, including complex SQL operations like aggregation, grouping, ordering, and cross-modal retrieval. Use when the user wants to benchmark on SPARTA, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.23286
  bibtex_key: park2026sparta
  confidence: high
---

# sparta-eval

> SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables — Park et al. (2026) (arXiv:2602.23286, 2026)

## What this evaluates

Evaluates the ability of LLMs and hybrid QA systems to perform tree-structured, multi-hop reasoning over combined text and table data, including complex SQL operations like aggregation, grouping, ordering, and cross-modal retrieval.

## Datasets

- **SPARTA** — total ?; splits: test (-1); repo https://github.com/pshlego/SPARTA

## Metrics

- `F1` **(primary)** — range: percent
  - Harmonic mean of precision and recall over normalized answer matches. Precision is the fraction of predicted answers that match gold, recall is the fraction of gold answers matched by predictions.

## Input / output format

**Input**: A natural language question, optionally accompanied by a set of relevant tables and text passages (Oracle setting) or provided with no evidence (Retrieval setting, requiring the model to first retrieve relevant tables/text).

**Output**: A natural language answer or executable query/response. The system evaluates the final answer string against gold answers using exact match and F1 scoring.

## Scoring recipe

```python
def compute_f1(preds, golds):
    em = sum(1 for p, g in zip(preds, golds) if normalize(p) == normalize(g)) / len(golds)
    p = sum(1 for p, g in zip(preds, golds) if normalize(p) in normalize(g)) / len(golds)
    r = sum(1 for p, g in zip(preds, golds) if normalize(g) in normalize(p)) / len(golds)
    f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
    return em, f1, p, r
```

## Common pitfalls

- Models are evaluated under two distinct settings: Oracle (given gold tables/text) vs. Retrieval (must find evidence first). Performance drops significantly in the Retrieval setting.
- Standard single-hop QA benchmarks do not capture the difficulty of tree-structured multi-hop queries with aggregations/grouping, leading to misleadingly high baseline scores on older datasets.
- Table size matters: domains with thousands of rows (e.g., NBA, Movie) cause sharper performance degradation than smaller tables.

## Evidence (verbatim from paper)

> On SPARTA (Oracle), ODYSSEY with GPT-5 achieves an average F1 score of 35.6% across all domains, representing a sharp 33.9-point drop compared to its performance on HybridQA (69.5%). We evaluate SPARTA under two configurations: (1) SPARTA (Oracle), where models are given ground-truth tables and linked passages; and (2) SPARTA (Retrieval), where models must retrieve relevant content from the entire corpus.

## Citation

```bibtex
@misc{park2026sparta,
  title={SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables},
  author={Park et al. (2026)},
  year={2026},
  note={arXiv:2602.23286}
}
```

- arXiv: 2602.23286