Spartqa Eval

---
name: spartqa-eval
description: Probes a model's ability to perform multi-step spatial reasoning over natural language stories. It evaluates understanding of spatial relations (e.g., near, far, containment) and tests robustness against surface-level vocabulary changes and question phrasing variations. Use when the user wants to benchmark on SPARTQA-HUMAN, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.05832
  bibtex_key: mirzaee2021spartqa
  confidence: high
---

# spartqa-eval

> SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning — Mirzaee et al. (2021) (arXiv:2104.05832, 2021)

## What this evaluates

Probes a model's ability to perform multi-step spatial reasoning over natural language stories. It evaluates understanding of spatial relations (e.g., near, far, containment) and tests robustness against surface-level vocabulary changes and question phrasing variations.

## Datasets

- **SPARTQA-HUMAN** — total 1100; splits: train (600), test (500)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Micro-average accuracy across four question types (FB, FR, CO, YN). Computed as the proportion of correctly predicted answers over the total number of test instances.
- `F1 score` — range: [0, 1]
  - Binary F1 score computed specifically for the Yes/No (YN) question type to account for class imbalance.
- `consistency & contrast accuracy` — range: [0, 1]
  - Accuracy on modified question sets (consistency: same answer; contrast: flipped answer), evaluated only on questions already predicted correctly by the model.

## Input / output format

**Input**: A contextual story paragraph followed by a multiple-choice question targeting spatial relations.

**Output**: Predicted answer choice (string or class label corresponding to the options).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(p == g for p, g in zip(predictions, gold_labels))
    return correct / len(gold_labels)

def compute_f1_binary(predictions, gold_labels):
    tp = sum(p == g == 1 for p, g in zip(predictions, gold_labels))
    fp = sum(p == 1 and g == 0 for p, g in zip(predictions, gold_labels))
    fn = sum(p == 0 and g == 1 for p, g in zip(predictions, gold_labels))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- Excluded 'near to' and 'far from' relations from evaluation due to high annotator variance.
- High accuracy on the 'Seen' test set often reflects vocabulary overfitting rather than true spatial reasoning; the 'Unseen' set is required to verify generalization.
- Yes/No (YN) questions are highly imbalanced, making accuracy misleading; F1 is required for fair evaluation.
- Consistency and contrast metrics are only computed on questions already predicted correctly, which can overestimate model reliability.

## Evidence (verbatim from paper)

> Table 2 shows performance on SPARTQA-HUMAN in a low-resource setting, where 0.6k QA pairs from SPARTQA-HUMAN are used for fine-tuning these LMs and 0.5k for testing (see Table 1 for information on this split). To verify it, we compute the F1 score for YN Q-TYPE in Table 3, where we see all systems effectively achieve better scores than the majority baseline.

## Citation

```bibtex
@misc{mirzaee2021spartqa,
  title={SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning},
  author={Mirzaee et al. (2021)},
  year={2021},
  note={arXiv:2104.05832}
}
```

- arXiv: 2104.05832