Clash Eval

---
name: clash-eval
description: This benchmark probes multimodal large language models' ability to detect cross-modal contradictions between images and text. It evaluates whether models can identify inconsistencies when either modality contains errors or hallucinations, rather than assuming one modality is ground truth. The task reveals systematic modality biases and category-specific reasoning weaknesses. Use when the user wants to benchmark on CLASH, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.19199
  bibtex_key: popordanoska2025clash
  confidence: high
---

# clash-eval

> CLASH: A Benchmark for Cross-Modal Contradiction Detection — Popordanoska et al. (2025) (arXiv:2511.19199, 2025)

## What this evaluates

This benchmark probes multimodal large language models' ability to detect cross-modal contradictions between images and text. It evaluates whether models can identify inconsistencies when either modality contains errors or hallucinations, rather than assuming one modality is ground truth. The task reveals systematic modality biases and category-specific reasoning weaknesses.

## Datasets

- **CLASH** — total 16289; splits: train (15000), test (1289)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly identified contradictory elements. For multiple-choice tasks, accuracy is calculated by selecting the 'Conflicting information – cannot answer' option. For open-ended tasks, accuracy is determined by whether the generated response correctly identifies the contradiction.

## Input / output format

**Input**: An image (from MS COCO), a contradictory text caption, and a targeted question focusing on the conflicting element.

**Output**: Multiple-choice: selection of one of four options (image-grounded, text-grounded, plausible distractor, or 'Conflicting information – cannot answer'). Open-ended: free-form text response identifying the contradiction.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        if pred == gold_ans:
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Models often exhibit modality bias, defaulting to trusting either the image or the text rather than detecting the conflict.
- Open-ended generation tasks show significantly lower performance than multiple-choice, indicating models struggle to articulate contradictions without constrained options.
- Category-specific failures (e.g., confusing object vs. attribute contradictions) can skew aggregate accuracy if not analyzed separately.

## Evidence (verbatim from paper)

> revealing systematic modality biases and category-specific weaknesses—especially in open-source models—while demonstrating that targeted fine-tuning significantly improves conflict detection performance (e.g., LLaVA-1.5-7b improves from 0% to 77% accuracy).

## Citation

```bibtex
@misc{popordanoska2025clash,
  title={CLASH: A Benchmark for Cross-Modal Contradiction Detection},
  author={Popordanoska et al. (2025)},
  year={2025},
  note={arXiv:2511.19199}
}
```

- arXiv: 2511.19199