Shot2story Eval

---
name: shot2story-eval
description: Evaluates multi-modal video understanding across three tasks: single-shot captioning, multi-shot summarization, and question answering. It probes a model's ability to process visual frames, optional ASR text, and shot-level structure to generate coherent descriptions or answer temporal, holistic, and audio-related questions. Use when the user wants to benchmark on Shot2Story, or asks about evaluating this task. Reports CIDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.10300
  bibtex_key: han2023shot2story
  confidence: high
---

# shot2story-eval

> Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos — Han et al. (2023) (arXiv:2312.10300, 2023)

## What this evaluates

Evaluates multi-modal video understanding across three tasks: single-shot captioning, multi-shot summarization, and question answering. It probes a model's ability to process visual frames, optional ASR text, and shot-level structure to generate coherent descriptions or answer temporal, holistic, and audio-related questions.

## Datasets

- **Shot2Story** — total 42958; splits: train (36951), val (1982), test (4025); repo https://github.com/bytedance/Shot2Story

## Metrics

- `BLEU@4` — range: [0, 1]
  - 4-gram precision with brevity penalty. Measures exact n-gram overlap between predicted and reference text.
- `METEOR` — range: [0, 1]
  - Harmonic mean of unigram precision and recall, weighted by penalty for fragmentation and synonym/stem matching.
- `ROUGE` — range: [0, 1]
  - Recall-oriented overlap of n-grams (typically ROUGE-L) between hypothesis and reference.
- `CIDEr` **(primary)** — range: other
  - Consensus-based Image Description Evaluation. Computes TF-IDF weighted n-gram similarity between predicted and reference captions to emphasize consensus with human references.
- `Accuracy` — range: percent
  - Percentage of correctly answered questions. Correctness is determined by a binary decision from GPT-3.5-turbo comparing the model's answer to the ground truth.

## Input / output format

**Input**: Video frames sampled and resized to 224x224, optionally concatenated with ASR text transcripts. For QA tasks, the generated summary and a natural language question are provided as text prompts to an LLM.

**Output**: Text output: either a single-shot caption, a multi-shot video summary paragraph, or a direct answer to a question.

## Scoring recipe

```python
def score_captioning(pred, gold):
    bleu4 = compute_bleu(gold, pred, max_order=4)
    meteor = compute_meteor(gold, pred)
    rouge = compute_rouge_l(gold, pred)
    cider = compute_cider(gold, pred)
    return bleu4, meteor, rouge, cider

def score_qa(pred, gold, question):
    prompt = f'Q: {question}\nGold: {gold}\nPred: {pred}\nIs pred correct? (yes/no)'
    judge_response = llm_call('gpt-3.5-turbo', prompt)
    is_correct = 1 if 'yes' in judge_response.lower() else 0
    return is_correct

# Aggregate
accuracy = mean([score_qa(p, g, q) for p, g, q in qa_pairs])
```

## Common pitfalls

- Confusing raw audio input with ASR text: the paper explicitly notes that ASR text is critical for performance, but adding raw audio features yields negligible gains.
- Overlooking shot-level structure: holistic models that ignore explicit shot boundaries underperform compared to shot-aware architectures, making shot processing essential for multi-shot tasks.
- QA evaluation relies on GPT-3.5-turbo for binary correctness rather than exact match, which may introduce judge bias or leniency compared to standard string-matching metrics.

## Evidence (verbatim from paper)

> For captioning and summarization, we evaluate the models using BLEU@4 (Papineni et al., 2002) (B), METEOR (Denkowski & Lavie, 2014) (M), ROUGE (Lin, 2004) (R), and CIDEr (Vedantam et al., 2015) (C).

## Citation

```bibtex
@misc{han2023shot2story,
  title={Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos},
  author={Han et al. (2023)},
  year={2023},
  note={arXiv:2312.10300}
}
```

- arXiv: 2312.10300