Vista Eval

---
name: vista-eval
description: This benchmark evaluates a model's ability to generate concise, structured summaries of scientific conference talks from video inputs. It specifically probes informativeness, alignment with visual/audio content, and factual consistency against the corresponding paper abstracts. Use when the user wants to benchmark on VISTA, or asks about evaluating this task. Reports ROUGE-1 F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.08279
  bibtex_key: liu2025vista
  confidence: high
---

# vista-eval

> What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentations — Liu et al. (2025) (arXiv:2502.08279, 2025)

## What this evaluates

This benchmark evaluates a model's ability to generate concise, structured summaries of scientific conference talks from video inputs. It specifically probes informativeness, alignment with visual/audio content, and factual consistency against the corresponding paper abstracts.

## Datasets

- **VISTA** — total 18599; splits: test (-1)

## Metrics

- `ROUGE-1 F1` **(primary)** — range: [0, 1]
  - F1 score of unigram overlap between the generated summary and the reference abstract.
- `ROUGE-2 F1` — range: [0, 1]
  - F1 score of bigram overlap between the generated summary and the reference abstract.
- `ROUGE-LSum F1` — range: [0, 1]
  - Summary-level ROUGE F1 score that treats each paragraph as a separate document.
- `SacreBLEU` — range: [0, 100]
  - BLEU score computed with standardized tokenization and a fixed reference corpus.
- `METEOR` — range: [0, 1]
  - Alignment-based metric incorporating synonymy, stemming, and word order penalties.
- `BERTScore` — range: [0, 1]
  - Cosine similarity between contextual embeddings of generated and reference tokens, aggregated via F1.
- `CIDEr-D` — range: [0, 1]
  - TF-IDF weighted n-gram similarity score designed for image and video captioning evaluation.
- `VideoScore` — range: [0, 1]
  - Metric evaluating how well the generated summary aligns with the input video content.
- `FactVC` — range: [0, 1]
  - Metric measuring factual consistency between the generated summary and the source video/abstract.

## Input / output format

**Input**: Video files (containing visual and audio modalities), optionally pre-processed into transcripts or OCR text, accompanied by standardized prompt instructions.

**Output**: A text summary of the scientific presentation, intended to align with the structure and content of the corresponding paper abstract.

## Scoring recipe

```python
def compute_metrics(predictions, references, videos):
    scores = {}
    scores['ROUGE-1 F1'] = rouge_score(references, predictions, 'rouge1').fmeasure
    scores['ROUGE-2 F1'] = rouge_score(references, predictions, 'rouge2').fmeasure
    scores['ROUGE-LSum F1'] = rouge_score(references, predictions, 'rougeLsum').fmeasure
    scores['SacreBLEU'] = sacrebleu.corpus_bleu(predictions, [references]).score
    scores['METEOR'] = meteor_score(references, predictions)
    scores['BERTScore'] = bertscore_score(references, predictions, lang='en').f1.mean()
    scores['CIDEr-D'] = cider_score(references, predictions)
    scores['VideoScore'] = videoscore_model.evaluate(predictions, videos)
    scores['FactVC'] = factvc_model.evaluate(predictions, videos, references)
    return scores
```

## Common pitfalls

- Models may inadvertently rely solely on audio transcripts or OCR text rather than multimodal video features, skewing alignment scores if not explicitly controlled.
- Factual consistency is particularly challenging for technical scientific content where visual slides contain critical data not present in the audio or abstract.
- Zero-shot, QLoRA, and full fine-tuning settings produce vastly different baselines; comparing them without identical hyperparameter settings leads to unfair benchmarking.

## Evidence (verbatim from paper)

> We report a set of evaluation metrics to measure informativeness, alignment, and factual consistency in summaries. For informativeness, we utilize ROUGE, SacreBLEU, METEOR, BERTScore, and CIDEr-D. Specifically, we provide the F1 scores for Rouge-1 (R1), Rouge-2 (R2), and Rouge-LSum (RLSUM). Alignment to the input video is evaluated with VideoScore, and factual consistency with FactVC.

## Citation

```bibtex
@misc{liu2025vista,
  title={What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentations},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2502.08279}
}
```

- arXiv: 2502.08279