Hal Eval Eval

---
name: hal-eval-eval
description: Evaluates Large Vision-Language Models (LVLMs) for hallucinations by measuring their ability to generate faithful image descriptions (generative evaluation) and detect hallucinations in provided captions (discriminative evaluation). It specifically probes fine-grained hallucination categories: object, relation, attribute, and event hallucinations, while also analyzing the impact of output length and Chain-of-Thought prompting. Use when the user wants to benchmark on COCO 2014, or asks about evaluating this task. Reports F1 score, Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.15721
  bibtex_key: jiang2024haleval
  confidence: high
---

# hal-eval-eval

> Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models — Chaoya Jiang et al. (arXiv:2402.15721, 2024)

## What this evaluates

Evaluates Large Vision-Language Models (LVLMs) for hallucinations by measuring their ability to generate faithful image descriptions (generative evaluation) and detect hallucinations in provided captions (discriminative evaluation). It specifically probes fine-grained hallucination categories: object, relation, attribute, and event hallucinations, while also analyzing the impact of output length and Chain-of-Thought prompting.

## Datasets

- **COCO 2014** — total 10000; splits: in-domain (5000), out-of-domain (5000)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for detecting hallucinations per type (object, relation, attribute, event) in discriminative evaluation. Calculated as 2 * (precision * recall) / (precision + recall).
- `Accuracy` **(primary)** — range: percent
  - Percentage of generated image descriptions that contain zero hallucinations according to the AFHA annotation framework in generative evaluation.
- `Hallucination Ratio` — range: percent
  - Proportion of generated outputs that contain a specific hallucination type (object, relation, attribute, or event) out of the total evaluated outputs.
- `Pearson's r / Spearman's ρ / Kendall's τ` — range: [-1, 1]
  - Statistical correlation coefficients measuring the alignment between automated metric scores and human judgment on hallucination presence across a 50-image test set.

## Input / output format

**Input**: Image paired with a text prompt for generation; for discriminative evaluation, image paired with a reference caption or question.

**Output**: Text description of the image (generative); binary classification or hallucination type label (discriminative).

## Scoring recipe

```python
def score_discriminative(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'hallucination' and g == 'hallucination')
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'hallucination' and g == 'no')
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p == 'no' and g == 'hallucination')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

def score_generative(predictions, gold_captions):
    correct = sum(1 for pred in predictions if not contains_hallucination(pred, gold_captions))
    accuracy = correct / len(predictions)
    hallucination_ratio = sum(1 for pred in predictions if has_type_hallucination(pred, target_type)) / len(predictions)
    return accuracy, hallucination_ratio
```

## Common pitfalls

- Models exhibit a strong 'yes' bias in discriminative settings, which can artificially inflate accuracy if not mitigated via specific prompting or filtering.
- Event hallucinations are significantly harder to detect than object/attribute hallucinations, especially in generative settings with long outputs, leading to misleadingly high discriminative scores.
- Output length strongly correlates with hallucination rates; evaluating only short outputs masks the model's true faithfulness on complex descriptions.

## Evidence (verbatim from paper)

> Table [4] displays the average F1 scores for various hallucination types in discriminative assessments and the average hallucination rates for LVLM’s long outputs (>=80) in generative assessments.

## Citation

```bibtex
@misc{jiang2024haleval,
  title={Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models},
  author={Chaoya Jiang et al.},
  year={2024},
  note={arXiv:2402.15721}
}
```

- arXiv: 2402.15721