Socnern Gar Eval

---
name: socnern-gar-eval
description: This benchmark evaluates a model's ability to recognize coordinated group activities in soccer matches by comparing two input modalities: raw video pixels and structured positional tracking data. It probes spatial-temporal reasoning, tactical formation understanding, and robustness to visual shifts by measuring how well models classify 10 distinct group actions from synchronized match footage. Use when the user wants to benchmark on SoccerNet-GAR, or asks about evaluating this task. Reports balanced accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.12606
  bibtex_key: karki2025pixels
  confidence: high
---

# socnern-gar-eval

> Pixels or Positions? Benchmarking Modalities in Group Activity Recognition — Karki et al. (2025) (arXiv:2511.12606, 2025)

## What this evaluates

This benchmark evaluates a model's ability to recognize coordinated group activities in soccer matches by comparing two input modalities: raw video pixels and structured positional tracking data. It probes spatial-temporal reasoning, tactical formation understanding, and robustness to visual shifts by measuring how well models classify 10 distinct group actions from synchronized match footage.

## Datasets

- **SoccerNet-GAR** — total 94285; splits: test (14519), train (-1), val (-1)

## Metrics

- `balanced accuracy` **(primary)** — range: percent
  - Average per-class recall across all 10 activity classes. For each class, recall is computed as TP / (TP + FN). The final score is the arithmetic mean of these 10 recall values, mitigating bias from severe class imbalance.
- `per-class accuracy` — range: percent
  - Standard accuracy computed independently for each of the 10 activity classes (TP / (TP + FP + FN)). Used for detailed ablation and modality comparison.

## Input / output format

**Input**: Two modalities are evaluated: (1) Tracking: frame-level graphs with 23 nodes (8-dimensional entity features) and positional role-based edges (goalkeeper, defender, midfielder, forward). (2) Video: 224×224×3 RGB video frames. Both modalities process synchronized sequences from FIFA World Cup 2022 matches.

**Output**: A single class label from the 10-class vocabulary: PASS, TACKLE, OUT, HEADER, HIGH PASS, THROW IN, CROSS, FREE KICK, SHOT, GOAL.

## Scoring recipe

```python
def compute_balanced_accuracy(predictions, gold_labels, num_classes=10):
    recalls = []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        recalls.append(recall)
    return sum(recalls) / num_classes
```

## Common pitfalls

- Severe class imbalance (e.g., GOAL has only 30 test samples) makes standard accuracy misleading; balanced accuracy must be used.
- Confusion between spatially similar actions (e.g., HIGH PASS vs PASS) requires careful per-class analysis rather than relying solely on aggregate scores.
- Tracking models use sentinel coordinates (-2.0) for missing entities, which must be excluded from graph message passing to avoid corrupting representations.

## Evidence (verbatim from paper)

> Metrics. We report balanced accuracy (average per-class recall) as the primary metric due to severe class imbalance.

## Citation

```bibtex
@misc{karki2025pixels,
  title={Pixels or Positions? Benchmarking Modalities in Group Activity Recognition},
  author={Karki et al. (2025)},
  year={2025},
  note={arXiv:2511.12606}
}
```

- arXiv: 2511.12606