Contamination Rate

---
name: contamination_rate
description: Measures the extent to which multimodal evaluation benchmarks are contaminated by pre-training data, assessing both visual similarity and textual inference leakage to quantify data contamination risks. Use when the user has predictions and gold and needs to compute image-only contamination rate.
metadata:
  skill_kind: metric
  source_arxiv: 2410.08695
  bibtex_key: yang2024dynamicmultimodalevaluation
  confidence: medium
---

# contamination_rate

> Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping — Yang et al. (2024) (arXiv:2410.08695, 2024)

## What this evaluates

Measures the extent to which multimodal evaluation benchmarks are contaminated by pre-training data, assessing both visual similarity and textual inference leakage to quantify data contamination risks.

## Datasets

- **SEEDBench** — total ?; splits: eval (-1)
- **MMBench** — total ?; splits: eval (-1)
- **MME** — total ?; splits: eval (-1)

## Metrics

- `image-only contamination rate` **(primary)** — range: [0, 1]
  - Ratio of evaluation images with a CLIPScore exceeding 0.9 against any pre-training image to the total number of evaluation images.
- `image-text contamination rate` — range: [0, 1]
  - Ratio of evaluation samples where the answer can be directly inferred from pre-training image captions (detected via GPT-4) to the total number of evaluation samples.

## Input / output format

**Input**: Evaluation benchmark images and their corresponding questions/answers, compared against pre-training dataset images and captions.

**Output**: A binary contamination flag per image or image-text pair, aggregated into a contamination rate.

## Scoring recipe

```python
def calculate_contamination(eval_images, eval_samples, train_images, train_captions):
    contaminated_img_count = 0
    for img in eval_images:
        if max(clip_score(img, t_img) for t_img in train_images) > 0.9:
            contaminated_img_count += 1
    img_only_rate = contaminated_img_count / len(eval_images)
    
    contaminated_pair_count = 0
    for q, a, img in eval_samples:
        if gpt4_inferred(a, train_captions):
            contaminated_pair_count += 1
    img_text_rate = contaminated_pair_count / len(eval_samples)
    return img_only_rate, img_text_rate
```

## Common pitfalls

- The CLIPScore threshold of 0.9 is a heuristic and may miss semantically similar but visually distinct images.
- GPT-4 inference for text contamination is non-deterministic and heavily depends on the prompt design used for caption comparison.
- Analysis is limited to the maximum pre-training dataset size checked (100M), which underestimates actual contamination in modern large-scale models.

## Evidence (verbatim from paper)

> We adopt 0.9 as the threshold to determine visual contamination. The image-only contamination rate is calculated as the ratio of the number of contaminated images and the number of total images in the evaluation set.

## Citation

```bibtex
@misc{yang2024dynamicmultimodalevaluation,
  title={Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2410.08695}
}
```

- arXiv: 2410.08695