Pistachio Eval

---
name: pistachio-eval
description: Evaluates video anomaly detection and understanding on synthetic, long-form videos with diverse scenes and balanced anomaly categories. Probes models' temporal consistency, ability to detect subtle behavioral anomalies, and long-context narrative comprehension. Use when the user wants to benchmark on Pistachio, or asks about evaluating this task. Reports frame-level AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.19474
  bibtex_key: li2025pistachio
  confidence: high
---

# pistachio-eval

> Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmarks — Jie Li et al. (2025) (arXiv:2511.19474, 2025)

## What this evaluates

Evaluates video anomaly detection and understanding on synthetic, long-form videos with diverse scenes and balanced anomaly categories. Probes models' temporal consistency, ability to detect subtle behavioral anomalies, and long-context narrative comprehension.

## Datasets

- **Pistachio** — total 1676822; splits: train (1544546), test (132276)

## Metrics

- `frame-level AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve computed per frame to measure detection performance.
- `frame-level AP` — range: [0, 1]
  - Average Precision computed per frame, summarizing the precision-recall curve across confidence thresholds.
- `F1-Score` — range: [0, 1]
  - Harmonic mean of precision and recall. Precision and recall are calculated by comparing generated anomaly descriptions against ground truth, counting a prediction as correct if semantic similarity exceeds a predefined threshold.

## Input / output format

**Input**: Video frames or clips for detection; full video for understanding.

**Output**: Per-frame anomaly scores/labels for detection; structured anomaly descriptions/narratives for understanding.

## Scoring recipe

```python
# VAD
auc = compute_auc(y_true_frames, y_pred_scores)
ap = compute_ap(y_true_frames, y_pred_scores)
# VAU
correct = sum(1 for pred, gt in zip(predictions, ground_truth) if semantic_similarity(pred, gt) > threshold)
precision = correct / len(predictions)
recall = correct / len(ground_truth)
f1 = 2 * (precision * recall) / (precision + recall)
```

## Common pitfalls

- Models relying on likelihood-based normal distribution learning fail when normal video diversity is high, mistaking diverse normal scenes for anomalies.
- Long-form generation baselines suffer from prompt forgetting and ghosting artifacts when not using hierarchical prompting, leading to semantic drift over time.
- Backbone choice significantly impacts results: I3D features often outperform ViT features for specific anomaly categories despite ViT having higher overall AUC.

## Evidence (verbatim from paper)

> Following standard evaluation protocols, all methods were trained on the Pistachio training set and evaluated on our test set. Performance is measured by frame-level AUC and AP. To evaluate the benchmarks of video anomaly understanding, we adopt F1-Score as the main metric to measure the model’s capability in comprehending anomalies across different temporal granularities. The F1-Score is calculated as the harmonic mean of precision and recall, providing a balanced assessment of the model’s performance. For each level, we measure the F1-Score by comparing the model’s generated anomaly descriptions with ground truth annotations, considering a prediction as correct when the semantic similarity exceeds a predefined threshold.

## Citation

```bibtex
@misc{li2025pistachio,
  title={Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmarks},
  author={Jie Li et al. (2025)},
  year={2025},
  note={arXiv:2511.19474}
}
```

- arXiv: 2511.19474