Vc Inspector

---
name: vc-inspector
description: Evaluates the factual accuracy and overall quality of video captions in a reference-free setting. It measures how well a model's predicted quality scores and explanations align with human judgments across diverse video and image domains. Use when the user has predictions and gold and needs to compute Kendall's correlation ($\tau_b$).
metadata:
  skill_kind: metric
  source_arxiv: 2509.16538
  bibtex_key: [REDACTED]
  confidence: high
---

# vc-inspector

> VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analy — Shubhashis Roy Dipta et al. (2025) (arXiv:2509.16538, 2025)

## What this evaluates

Evaluates the factual accuracy and overall quality of video captions in a reference-free setting. It measures how well a model's predicted quality scores and explanations align with human judgments across diverse video and image domains.

## Datasets

- **VATEX-EVAL** — total 15540; splits: test (15540)
- **ActivityNet-FG-Eval** — total ?; splits: eval (-1)
- **YouCook2-FG-Eval** — total ?; splits: eval (-1)
- **Flickr8K-Expert** — total 17000; splits: test (17000)
- **Flickr8K-CF** — total 48000; splits: test (48000)

## Metrics

- `Kendall's correlation ($\tau_b$)` **(primary)** — range: [-1, 1] (reported as 0–100% in tables)
  - Measures the rank correlation between predicted caption quality scores and human-annotated scores. Adjusts for ties in both predicted and gold scores to provide a robust ordinal correlation estimate.
- `Spearman's rank correlation ($\rho$)` — range: [-1, 1] (reported as 0–100% in tables)
  - Measures the monotonic relationship between predicted and human-annotated quality scores based on rank differences. Computed as 1 - (6 * sum(d_i^2)) / (n * (n^2 - 1)).

## Input / output format

**Input**: Video frames (uniformly sampled to 32 frames, resized to 224x224) or a single image, paired with a candidate caption.

**Output**: A quality score on a 1–5 scale and a textual explanation identifying factual errors in the caption.

## Scoring recipe

```python
def compute_kendall_tau_b(pred_scores, gold_scores):
    n = len(pred_scores)
    concordant = discordant = 0
    for i in range(n):
        for j in range(i + 1, n):
            diff_p = pred_scores[i] - pred_scores[j]
            diff_g = gold_scores[i] - gold_scores[j]
            if diff_p * diff_g > 0: concordant += 1
            elif diff_p * diff_g < 0: discordant += 1
    n0 = n * (n - 1) / 2
    tau_b = (concordant - discordant) / n0
    return tau_b * 100  # Paper reports as percentage
```

## Common pitfalls

- The synthetic evaluation datasets (ActivityNet-FG-Eval, YouCook2-FG-Eval) use LLM-generated pseudo-scores, not direct human annotations; human alignment is only verified on VATEX-EVAL and Flickr8K.
- Correlation scores measure ranking alignment with human judgments, not absolute score accuracy; higher values indicate better relative ordering of caption quality.
- The metric is explicitly designed for the reference-free setting, but results are often compared against reference-based baselines, which can mislead readers about its actual input requirements.

## Evidence (verbatim from paper)

> Following prior work, we report both Kendall’s correlation ($\tau_{b}$) and Spearman’s rank correlation ($\rho$). In the caption evaluation literature, the effectiveness of evaluation metrics is typically assessed by measuring their correlation to ground truth.

## Citation

```bibtex
@misc{roydipta2025vcinspector,
  title={VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analy},
  author={Shubhashis Roy Dipta et al. (2025)},
  year={2025},
  note={arXiv:2509.16538}
}
```

- arXiv: 2509.16538