Fiova Eval

---
name: fiova-eval
description: Evaluates the ability of Large Vision-Language Models (LVLMs) to generate accurate, fluent, and comprehensive captions for long videos. It probes semantic alignment, event coverage, and hallucination resistance by comparing model outputs against multi-annotator human references. Use when the user wants to benchmark on FIOVA, or asks about evaluating this task. Reports FIOVA-DQ F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.15270
  bibtex_key: hu2024fiova
  confidence: high
---

# fiova-eval

> FIOVA: A Multi-Annotator Benchmark for Human-Aligned Video Captioning — Hu et al. (2024) (arXiv:2410.15270, 2024)

## What this evaluates

Evaluates the ability of Large Vision-Language Models (LVLMs) to generate accurate, fluent, and comprehensive captions for long videos. It probes semantic alignment, event coverage, and hallucination resistance by comparing model outputs against multi-annotator human references.

## Datasets

- **FIOVA** — total 3002; splits: test (-1)

## Metrics

- `BLEU` — range: [0, 1]
  - Standard n-gram overlap metric measuring exact lexical matches between prediction and ground truth.
- `METEOR` — range: [0, 1]
  - Metric that aligns predictions and references using stemming, synonymy, and exact matches, weighting precision and recall.
- `GLEU` — range: [0, 1]
  - Generative Language Understanding Evaluation; measures n-gram overlap with a penalty for repetition.
- `FIOVA-DQ F1` **(primary)** — range: [0, 1]
  - Event-level metric that extracts events from the model caption and ground truth, then computes F1 based on event overlap weighted by cognitive weights derived from multi-annotator consensus.
- `FIOVA-DQ Precision` — range: [0, 1]
  - Proportion of events in the model caption that are correctly identified and weighted by annotator consensus in the ground truth.
- `FIOVA-DQ Recall` — range: [0, 1]
  - Proportion of consensus-weighted events in the ground truth that are successfully captured by the model caption.

## Input / output format

**Input**: Raw video file (average duration 33.6 seconds)

**Output**: Natural language caption describing the video content

## Scoring recipe

```python
def score_video_caption(pred_caption, gt_caption):
    # Traditional metrics
    bleu = compute_bleu(pred_caption, gt_caption)
    meteor = compute_meteor(pred_caption, gt_caption)
    gleu = compute_gleu(pred_caption, gt_caption)
    
    # FIOVA-DQ event-level scoring
    events_pred = extract_events(pred_caption)
    events_gt = extract_events(gt_caption)  # aggregated from 5 annotators
    weights = compute_consensus_weights(events_gt)  # based on annotator agreement
    
    matched = count_weighted_matches(events_pred, events_gt, weights)
    precision = matched / max(len(events_pred), 1)
    recall = matched / max(len(events_gt), 1)
    f1 = 2 * (precision * recall) / (precision + recall + 1e-8)
    
    return {'BLEU': bleu, 'METEOR': meteor, 'GLEU': gleu, 
            'FIOVA-DQ_Precision': precision, 'FIOVA-DQ_Recall': recall, 'FIOVA-DQ_F1': f1}
```

## Common pitfalls

- Using a single human annotation as ground truth instead of the aggregated multi-annotator reference, which artificially lowers metric scores and misrepresents model performance.
- Relying solely on traditional n-gram metrics (BLEU/METEOR) which show near-zero correlation with human preference and event-level quality.
- Ignoring the cognitive weighting scheme in FIOVA-DQ, which adjusts for inter-subjective agreement rather than treating all events equally.

## Evidence (verbatim from paper)

> We computed Spearman correlations between the human rankings and automatic rankings derived from six metrics: F1, Precision, and Recall under FIOVA-DQ, and F1, Precision, and Recall under AutoDQ. Tab. A5 presents detailed results for each evaluator.

## Citation

```bibtex
@misc{hu2024fiova,
  title={FIOVA: A Multi-Annotator Benchmark for Human-Aligned Video Captioning},
  author={Hu et al. (2024)},
  year={2024},
  note={arXiv:2410.15270}
}
```

- arXiv: 2410.15270