Chirp Eval

---
name: chirp-eval
description: Evaluates the quality of open-ended, long-form responses from Vision-Language Models (VLMs) using pairwise preference ranking across five fine-grained criteria: overall preference, relevance, reasoning, hallucinations, and details. Use when the user wants to benchmark on CHIRP, or asks about evaluating this task. Reports pairwise_preference.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.09672
  bibtex_key: roger2025chirp
  confidence: high
---

# chirp-eval

> CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models — Roger et al. (2025) (arXiv:2501.09672, 2025)

## What this evaluates

Evaluates the quality of open-ended, long-form responses from Vision-Language Models (VLMs) using pairwise preference ranking across five fine-grained criteria: overall preference, relevance, reasoning, hallucinations, and details.

## Datasets

- **CHIRP** — total 104; splits: test (104)

## Metrics

- `pairwise_preference` **(primary)** — range: [0, 1]
  - Proportion of correct pairwise preferences selected by human annotators or VLM evaluators across five criteria. Calculated as the number of correct winner selections divided by the total number of evaluated matchups per criterion.

## Input / output format

**Input**: Image-question pairs (images generated via DALL·E 3) with open-ended prompts. Models generate long-form text responses.

**Output**: Open-ended text responses. For evaluation, pairs of responses are presented to human annotators or VLM evaluators for preference ranking per criterion.

## Scoring recipe

```python
def compute_preference_rate(predictions, gold_labels, criteria):
    total_correct = 0
    total_evaluated = 0
    for matchup in predictions:
        for criterion in criteria:
            if matchup.winner == gold_labels[matchup.id][criterion]:
                total_correct += 1
            total_evaluated += 1
    return total_correct / total_evaluated if total_evaluated > 0 else 0.0
```

## Common pitfalls

- Results rely on randomly sampled pairwise matchups (5 per question) due to budget constraints, so rankings may fluctuate with different random seeds.
- The benchmark uses pairwise preference rather than absolute scoring, making direct cross-benchmark comparisons require careful normalization (e.g., min-max scaling across benchmarks).
- Automated VLM proxies (e.g., GPT-4V) are evaluated under controlled reasoning prompts, which may not reflect standard zero-shot evaluation conditions.

## Evidence (verbatim from paper)

> By designing 104 thought-provoking, category-diverse questions and generating image-question pairs via DALL·E 3, CHIRP enables pairwise human and VLM-based preference evaluations across five quality criteria—overall preference, relevance, reasoning, hallucinations, and details—revealing significant performance disparities between models that static metrics miss.

## Citation

```bibtex
@misc{roger2025chirp,
  title={CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models},
  author={Roger et al. (2025)},
  year={2025},
  note={arXiv:2501.09672}
}
```

- arXiv: 2501.09672