Onlysports Benchmark Eval

---
name: onlysports-benchmark-eval
description: Evaluates a sports-domain language model's generation capability on sports-specific tasks and its zero-shot commonsense reasoning performance on general benchmarks. Use when the user wants to benchmark on OnlySports Benchmark, HellaSwag, PIQA, ARC-challenge, ARC-easy, or asks about evaluating this task. Reports OS-acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.00286
  bibtex_key: chen2024onlysportslm
  confidence: medium
---

# onlysports-benchmark-eval

> OnlySportsLM: Optimizing Sports-Domain Language Models with SOTA Performance under Billion Parameters — Chen et al. (2024) (arXiv:2409.00286, 2024)

## What this evaluates

Evaluates a sports-domain language model's generation capability on sports-specific tasks and its zero-shot commonsense reasoning performance on general benchmarks.

## Datasets

- **OnlySports Benchmark** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1); HF `hellaswag`
- **PIQA** — total ?; splits: test (-1); HF `piqa`
- **ARC-challenge** — total ?; splits: test (-1); HF `ai2_arc`
- **ARC-easy** — total ?; splits: test (-1); HF `ai2_arc`

## Metrics

- `OS-acc` **(primary)** — range: percent
  - Accuracy on sports domain generation tasks, calculated as the proportion of correctly generated or selected outputs.
- `OS-rel` — range: percent
  - Relation/extraction accuracy or score for sports domain tasks.
- `OS-Avg.` — range: percent
  - Average score across OS-acc and OS-rel.
- `ARC-e` — range: percent
  - Accuracy on ARC-easy multiple-choice questions.
- `ARC-c` — range: percent
  - Accuracy on ARC-challenge multiple-choice questions.
- `PIQA` — range: percent
  - Accuracy on Physical Interaction Question Answering.
- `HS` — range: percent
  - Accuracy on HellaSwag completion task.

## Input / output format

**Input**: Zero-shot prompts for sports domain generation tasks and commonsense reasoning questions.

**Output**: Model-generated text completions or selected answer choices.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
    return correct / len(gold) * 100

# For multiple-choice benchmarks (ARC, PIQA, HS):
# predictions are model-generated choices or extracted answers.
# gold are ground-truth labels.
# Score is percentage of exact matches.
```

## Common pitfalls

- Baselines' general benchmark scores are retrieved directly from their original papers rather than re-evaluated on the same setup.
- Training was truncated at 315B tokens due to funding constraints, so results do not reflect convergence on the full 600B-token dataset.
- Evaluation is strictly zero-shot without instruction tuning or fine-tuning, which may disadvantage models trained with instruction data.

## Evidence (verbatim from paper)

> We compare the final OnlySportsLM checkpoint on OnlySports Benchmark and zero-shot commonsense reasoning tasks (Hellaswag, PIQA, ARC-challenge, and ARC-easy) with previous training checkpoints and recent open-source models. Table 3 compares our OnlySportsLM and two recent state-of-the-art general-purpose models, ranging from 137M to 1.7B parameters... OS-acc | OS-rel | OS-Avg. | ARC-e | ARC-c | PIQA | HS

## Citation

```bibtex
@misc{chen2024onlysportslm,
  title={OnlySportsLM: Optimizing Sports-Domain Language Models with SOTA Performance under Billion Parameters},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2409.00286}
}
```

- arXiv: 2409.00286