Marine Hallucination Eval

---
name: marine-hallucination-eval
description: Evaluates the ability of Large Vision-Language Models (LVLMs) to mitigate object hallucinations during text generation. It probes visual-text alignment by measuring hallucination rates, recall of existing objects, and accuracy on binary probing questions, alongside GPT-4V-aided assessments of response accuracy and detailness. Use when the user wants to benchmark on MSCOCO val2014, or asks about evaluating this task. Reports CHAIRS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.08680
  bibtex_key: zhao2024marine
  confidence: high
---

# marine-hallucination-eval

> Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance — Linxi Zhao et al. (2024) (arXiv:2402.08680, 2024)

## What this evaluates

Evaluates the ability of Large Vision-Language Models (LVLMs) to mitigate object hallucinations during text generation. It probes visual-text alignment by measuring hallucination rates, recall of existing objects, and accuracy on binary probing questions, alongside GPT-4V-aided assessments of response accuracy and detailness.

## Datasets

- **MSCOCO val2014** — total 5000; splits: val (5000)

## Metrics

- `CHAIRI` — range: [0, 1]
  - Instance-level hallucination rate: ratio of hallucinated objects to all mentioned objects in the generated caption.
- `CHAIRS` **(primary)** — range: [0, 1]
  - Sentence-level hallucination rate: ratio of captions containing hallucinated objects to all generated captions.
- `Recall` — range: [0, 1]
  - Instance-level recall: ratio of non-hallucinated (correctly mentioned) objects to all existing ground-truth objects.
- `POPE Accuracy` — range: [0, 1]
  - Accuracy on binary yes/no object probing questions in the adversarial setting.
- `POPE F1` — range: [0, 1]
  - F1 score for yes/no object probing responses in the adversarial setting.
- `GPT-4V Accuracy` — range: [1, 10]
  - Score out of 10 from GPT-4V judge on how accurately the model describes the image.
- `GPT-4V Detailness` — range: [1, 10]
  - Score out of 10 from GPT-4V judge on the richness of necessary details in the response.

## Input / output format

**Input**: An input image paired with a text prompt (e.g., image captioning instruction or binary yes/no question).

**Output**: A generated text response (natural language description or 'yes'/'no' answer).

## Scoring recipe

```python
def score_chair(pred_objects, gold_objects):
    hallucinated = set(pred_objects) - set(gold_objects)
    chair_i = len(hallucinated) / len(pred_objects) if pred_objects else 0
    chair_s = 1.0 if hallucinated else 0.0
    recall = len(gold_objects - hallucinated) / len(gold_objects) if gold_objects else 0.0
    return chair_i, chair_s, recall

def score_pope(preds, golds):
    acc = sum(1 for p, g in zip(preds, golds) if p == g) / len(golds)
    f1 = f1_score(golds, preds, pos_label='yes')
    yes_ratio = sum(1 for p in preds if p == 'yes') / len(preds)
    return acc, f1, yes_ratio

def score_gpt4v(image, prompt, pred):
    # Use GPT-4V judge prompt to get scores 1-10 for accuracy and detailness
    return gpt4v_judge(image, prompt, pred).accuracy, gpt4v_judge(image, prompt, pred).detailness
```

## Common pitfalls

- Fine-tuning baselines like LURE can overwrite original instructions and fail on binary question formats, making POPE evaluation impractical.
- POPE adversarial setting often suffers from a strong 'yes' bias in LVLMs, skewing accuracy and requiring explicit reporting of the yes-ratio.
- Excessive classifier-free guidance strength can degrade instruction-following capability despite reducing hallucinations, requiring careful tuning (γ∈(0.3,0.7)).

## Evidence (verbatim from paper)

> In alignment with established evaluations from previous studies, we assess our method using the following metrics: Caption Hallucination Assessment with Image Relevance (CHAIR)... Polling-based Object Probing Evaluation (POPE)... GPT-4V-aided Evaluation... All evaluations are conducted using the MSCOCO val2014 dataset.

## Citation

```bibtex
@misc{zhao2024marine,
  title={Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance},
  author={Linxi Zhao et al. (2024)},
  year={2024},
  note={arXiv:2402.08680}
}
```

- arXiv: 2402.08680