Soccerchat Eval

---
name: soccerchat-eval
description: Evaluates multimodal video-language models on soccer-specific tasks: referee decision validation via question-answering and multi-label action classification. It probes the model's ability to align visual, auditory, and textual cues with ground-truth soccer events and rules. Use when the user wants to benchmark on XFoul validation dataset, SoccerNet-v2, or asks about evaluating this task. Reports QwQ Scorer, F1 Score (wt).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.16630
  bibtex_key: gautam2025soccerchat
  confidence: medium
---

# soccerchat-eval

> SoccerChat: Integrating Multimodal Data for Enhanced Soccer Game Understanding — Gautam et al. (2025) (arXiv:2505.16630, 2025)

## What this evaluates

Evaluates multimodal video-language models on soccer-specific tasks: referee decision validation via question-answering and multi-label action classification. It probes the model's ability to align visual, auditory, and textual cues with ground-truth soccer events and rules.

## Datasets

- **XFoul validation dataset** — total ?; splits: test (-1)
- **SoccerNet-v2** — total ?; splits: test (-1)

## Metrics

- `QwQ Scorer` **(primary)** — range: [0, 10]
  - A custom LLM-based evaluator that assigns a score between 0 and 10 based on the alignment between the model's generated answer and the ground truth annotation.
- `F1 Score (wt)` **(primary)** — range: [0, 1]
  - Weighted F1-score for multi-label classification, averaging the F1 across classes weighted by their support (number of true instances).
- `Cohen Kappa` — range: [0, 1]
  - Statistical measure of inter-rater agreement for qualitative (categorical) items, adjusted for chance.
- `MCC` — range: [-1, 1]
  - Matthews Correlation Coefficient, a balanced measure of binary/multiclass classification quality that takes true/false positives and negatives into account.
- `Hamming Loss` — range: [0, 1]
  - Fraction of labels that are incorrectly predicted, averaged over all samples and labels.

## Input / output format

**Input**: Multimodal soccer match clips (video, audio, text) with jersey color annotations and ASR transcripts, paired with questions or event labels.

**Output**: Textual answers for QA tasks; class labels (6-class or 16-class) for action classification tasks.

## Scoring recipe

```python
# QwQ Scorer (QA/Referee Task)
# Returns float 0-10 based on LLM alignment with ground truth
score = qwq_model.evaluate(prediction=answer, gold=ground_truth)

# Action Classification Metrics (6-class or 16-class)
precision = precision_score(y_true, y_pred, average='weighted')
recall = recall_score(y_true, y_pred, average='weighted')
f1 = f1_score(y_true, y_pred, average='weighted')
kappa = cohen_kappa_score(y_true, y_pred)
mcc = matthews_corrcoef(y_true, y_pred)
hamming = hamming_loss(y_true, y_pred)
```

## Common pitfalls

- The QwQ Scorer is a custom LLM-based evaluator; exact scores depend on the specific model version and prompt template, making exact replication difficult without the inference code.
- Classification metrics use weighted averages, but the paper does not specify how multi-label predictions are thresholded or how class imbalance is handled during evaluation.
- Fine-tuning sequence (pretrain then fine-tune vs. joint training) drastically impacts performance, contrary to typical transfer learning expectations, requiring careful dataset mixing strategies.

## Evidence (verbatim from paper)

> The evaluation relied on the QwQ Scorer model, which assigns scores between 0 and 10 based on alignment with ground truth annotations. Performance comparisons were visualized using grouped violin plots to highlight score distributions across different model configurations.

## Citation

```bibtex
@misc{gautam2025soccerchat,
  title={SoccerChat: Integrating Multimodal Data for Enhanced Soccer Game Understanding},
  author={Gautam et al. (2025)},
  year={2025},
  note={arXiv:2505.16630}
}
```

- arXiv: 2505.16630