Video Star Eval

---
name: video-star-eval
description: Evaluates open-vocabulary action recognition by testing a model's ability to generalize to unseen action categories and cross-dataset distributions. It probes fine-grained video understanding and cross-modal reasoning capabilities under base-to-novel and cross-dataset generalization settings. Use when the user wants to benchmark on UCF-101, HMDB-51, Kinetics-400, Kinetics-600, Something-Something V2, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.08480
  bibtex_key: yuan2025videostar
  confidence: high
---

# video-star-eval

> Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools — Yuan et al. (2025) (arXiv:2510.08480, 2025)

## What this evaluates

Evaluates open-vocabulary action recognition by testing a model's ability to generalize to unseen action categories and cross-dataset distributions. It probes fine-grained video understanding and cross-modal reasoning capabilities under base-to-novel and cross-dataset generalization settings.

## Datasets

- **UCF-101** — total 13320; splits: train (-1), test (-1)
- **HMDB-51** — total 6849; splits: train (-1), test (-1)
- **Kinetics-400** — total ?; splits: train (-1), test (-1)
- **Kinetics-600** — total ?; splits: train (-1), test (-1)
- **Something-Something V2** — total ?; splits: train (-1), test (-1)

## Metrics

- `Top-1 accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted action classes out of the total number of test samples. Calculated as (number of correct predictions / total predictions) * 100.
- `Harmonic Mean (HM)` — range: percent
  - Harmonic mean of the Top-1 accuracy on base classes and novel classes. Calculated as 2 * (base_acc * novel_acc) / (base_acc + novel_acc). Used to balance performance across seen and unseen categories.

## Input / output format

**Input**: Raw video clips representing human actions, processed as visual tokens by the vision-language model.

**Output**: Predicted action class label.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, base_indices, novel_indices):
    top1 = sum(1 for p, g in zip(predictions, gold_labels) if p == g) / len(gold_labels) * 100
    base_acc = sum(1 for i in base_indices if predictions[i] == gold_labels[i]) / len(base_indices) * 100
    novel_acc = sum(1 for i in novel_indices if predictions[i] == gold_labels[i]) / len(novel_indices) * 100
    hm = 2 * (base_acc * novel_acc) / (base_acc + novel_acc) if (base_acc + novel_acc) > 0 else 0
    return {'top1_accuracy': top1, 'harmonic_mean': hm}
```

## Common pitfalls

- The base-to-novel setting evaluates on both base and novel classes, but the model is only trained on base classes, making it a zero-shot generalization setup rather than standard supervised evaluation.
- Baselines are fine-tuned on the base set of each respective dataset, whereas Video-STAR is only fine-tuned on HMDB-51's base set and evaluated zero-shot on others, creating an asymmetric comparison.
- Cross-dataset evaluation trains on a source dataset and tests on a target dataset, which measures transferability rather than in-distribution performance.

## Evidence (verbatim from paper)

> Following prior work(Rasheed et al., [2023]; Ni et al., [2022]; Weng et al., [2023]), we report average top-1 accuracy under the two settings.

## Citation

```bibtex
@misc{yuan2025videostar,
  title={Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools},
  author={Yuan et al. (2025)},
  year={2025},
  note={arXiv:2510.08480}
}
```

- arXiv: 2510.08480