Copo Hallucination Eval

---
name: copo-hallucination-eval
description: Evaluates the ability of Multimodal Large Language Models to generate factually grounded captions and answers while suppressing object-level hallucinations. It probes visual grounding, reasoning consistency, and alignment with human or GPT-4 preferences across multiple reasoning and perception benchmarks. Use when the user wants to benchmark on CHAIR, POPE, MMBench, MME, or asks about evaluating this task. Reports POPE F1 Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.04182
  bibtex_key: guo2025copo
  confidence: high
---

# copo-hallucination-eval

> COPO: Causal-Oriented Policy Optimization for Hallucinations of MLLMs — Peizheng Guo et al. (2025) (arXiv:2508.04182, 2025)

## What this evaluates

Evaluates the ability of Multimodal Large Language Models to generate factually grounded captions and answers while suppressing object-level hallucinations. It probes visual grounding, reasoning consistency, and alignment with human or GPT-4 preferences across multiple reasoning and perception benchmarks.

## Datasets

- **CHAIR** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)

## Metrics

- `CHAIR_I` — range: percent
  - Instance-level hallucination error rate: percentage of generated captions containing objects not present in the ground truth image.
- `CHAIR_S` — range: percent
  - Sentence-level hallucination error rate: percentage of sentences in generated captions containing hallucinated objects.
- `POPE F1 Score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall on object-level probing questions in a closed-set VQA format.
- `MMBench` — range: percent
  - Aggregate accuracy score across perception, grounding, and reasoning tasks.
- `MME` — range: percent
  - Aggregate score measuring perception, grounding, and reasoning capabilities.
- `GPT-4o Accuracy (A), Correctness (C), Detailedness (D)` — range: [1, 5]
  - Automatic ratings provided by GPT-4o on a scale for accuracy, correctness, and detailedness of generated captions.

## Input / output format

**Input**: Image paired with a text prompt (e.g., captioning instruction or VQA question).

**Output**: Generated text response (caption or answer).

## Scoring recipe

```python
def compute_metrics(predictions, golds, metric_name):
    if metric_name == 'CHAIR_I':
        return 100 * sum(1 for p, g in zip(predictions, golds) if has_hallucinated_objects(p, g)) / len(predictions)
    elif metric_name == 'CHAIR_S':
        return 100 * sum(1 for p, g in zip(predictions, golds) if any(has_hallucinated_objects(s, g) for s in p.split('.'))) / len(predictions)
    elif metric_name == 'POPE F1 Score':
        preds_labels = [1 if pred.lower() == 'yes' else 0 for pred in predictions]
        return f1_score(golds, preds_labels)
    elif metric_name == 'GPT-4o':
        return gpt4o_rate(predictions, rubric=['accuracy', 'correctness', 'detailedness'])
    return None
```

## Common pitfalls

- CHAIR error rates are lower-is-better, which is counterintuitive compared to standard accuracy metrics.
- POPE uses a closed-set probing VQA format rather than open-ended generation, so metrics reflect detection capability rather than generation quality.
- GPT-4o evaluation scores are subjective and depend heavily on the specific prompt template and temperature settings used during rating.

## Evidence (verbatim from paper)

> CHAIR measures object hallucinations in captioning by computing instance-level (CHAIR_I) and sentence-level (CHAIR_S) error rates, while POPE adopts a probing-based VQA setting to evaluate object-level hallucinations under controlled conditions. As shown in Table [1] and Table [2], our approach yields lower hallucination rates in CHAIR and higher F1 scores in POPE, demonstrating the effectiveness of our approach. For GPT-4 assisted evaluation, we compare model-generated captions on images, with GPT-4o rating them on accuracy (A), correctness (C), and detailedness (D).

## Citation

```bibtex
@misc{guo2025copo,
  title={COPO: Causal-Oriented Policy Optimization for Hallucinations of MLLMs},
  author={Peizheng Guo et al. (2025)},
  year={2025},
  note={arXiv:2508.04182}
}
```

- arXiv: 2508.04182