Triplesumm Video Summarization Eval

---
name: triplesumm-video-summarization-eval
description: Evaluates a model's ability to perform video summarization by predicting frame-level importance scores across visual, textual, and audio modalities. It probes the model's capacity for adaptive multimodal fusion and temporal dependency modeling to identify salient segments in long videos. Use when the user wants to benchmark on MoSu, Mr. HiSum, SumMe, TVSum, or asks about evaluating this task. Reports Kendall’s τ (kTau), Spearman’s ρ (sRho).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.01169
  bibtex_key: [REDACTED]
  confidence: high
---

# triplesumm-video-summarization-eval

> TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization — Sumin Kim et al. (2026) (arXiv:2603.01169, 2026)

## What this evaluates

Evaluates a model's ability to perform video summarization by predicting frame-level importance scores across visual, textual, and audio modalities. It probes the model's capacity for adaptive multimodal fusion and temporal dependency modeling to identify salient segments in long videos.

## Datasets

- **MoSu** — total ?; splits: original (-1); repo https://github.com/smkim37/TripleSumm
- **Mr. HiSum** — total ?; splits: original (-1)
- **SumMe** — total ?; splits: TV (-1), TVT (-1)
- **TVSum** — total ?; splits: TV (-1), TVT (-1)

## Metrics

- `Kendall’s τ (kTau)` **(primary)** — range: [-1, 1]
  - Rank-based correlation coefficient measuring the ordinal association between predicted frame-level importance scores and ground-truth importance labels.
- `Spearman’s ρ (sRho)` **(primary)** — range: [-1, 1]
  - Rank-based correlation coefficient assessing how well the predicted importance scores preserve the rank order of ground-truth importance.
- `mAP50` — range: [0, 1]
  - Mean Average Precision computed over the top 50% of 5-second segments ranked by predicted importance, evaluated against ground-truth highlights.
- `mAP15` — range: [0, 1]
  - Mean Average Precision computed over the top 15% of 5-second segments ranked by predicted importance, evaluated against ground-truth highlights.

## Input / output format

**Input**: Temporally aligned feature sequences of equal length for visual (CLIP frame embeddings at 1 fps), text (RoBERTa [CLS] tokens broadcast to frames), and audio (AST features at 1-sec intervals). For external datasets, official visual features plus generated captions and raw audio are used.

**Output**: Frame-level importance scores, which are subsequently ranked to identify salient segments.

## Scoring recipe

```python
def compute_metrics(pred_scores, gold_scores):
    # Rank-based correlations
    ktau = kendalltau(pred_scores, gold_scores)
    srho = spearmanr(pred_scores, gold_scores)
    # Segment-level highlight detection
    segments = split_video_into_segments(pred_scores, duration=5.0)
    top_50_idx = np.argsort(segments)[-int(len(segments)*0.5):]
    top_15_idx = np.argsort(segments)[-int(len(segments)*0.15):]
    mAP50 = compute_ap(top_50_idx, gold_scores)
    mAP15 = compute_ap(top_15_idx, gold_scores)
    return ktau, srho, mAP50, mAP15
```

## Common pitfalls

- SumMe and TVSum evaluations historically suffer from overfitting when using only train/validation splits; the paper explicitly compares against both TV and TVT protocols to correct this.
- mAP metrics are computed on 5-second segments, not raw frames, which changes the granularity of highlight detection compared to frame-level importance.
- The model must handle missing modalities gracefully; performance drops significantly if modality weighting isn't adaptive to content changes.

## Evidence (verbatim from paper)

> Following RethinkingSumm, we primarily report rank-based correlation metrics, Kendall’s τ (kTau) and Spearman’s ρ (sRho), on the frame-level importance prediction, being consistent with recent studies (CSTA; MAAM; SummDiff). On MoSu and Mr. HiSum datasets where the Most Replayed statistics are used as ground-truth, we further assess the highlight detection performance with mean Average Precision (mAP) following Mr.HiSum. Following this protocol, each video is divided into 5-second segments and ranked by predicted importance. The top 50% (mAP50) and 15% (mAP15) of these segments are then evaluated as the predicted highlights against the ground-truth.

## Citation

```bibtex
@misc{kim2026triplesumm,
  title={TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization},
  author={Sumin Kim et al. (2026)},
  year={2026},
  note={arXiv:2603.01169}
}
```

- arXiv: 2603.01169