Distortbench Eval

---
name: distortbench-eval
description: Evaluates vision-language models' ability to perform fine-grained low-level visual perception by identifying both the specific type of image distortion and its severity level from a single image. It probes whether models rely on direct perceptual pattern matching or struggle with subtle severity discrimination. Use when the user wants to benchmark on DistortBench, or asks about evaluating this task. Reports Acc..
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.19966
  bibtex_key: [REDACTED]
  confidence: high
---

# distortbench-eval

> DistortBench: Benchmarking Vision Language Models on Image Distortion Identification — Goyal et al. (2026) (arXiv:2604.19966, 2026)

## What this evaluates

Evaluates vision-language models' ability to perform fine-grained low-level visual perception by identifying both the specific type of image distortion and its severity level from a single image. It probes whether models rely on direct perceptual pattern matching or struggle with subtle severity discrimination.

## Datasets

- **DistortBench** — total 13500; splits: test (13500)

## Metrics

- `Acc.` **(primary)** — range: percent
  - Answered-only joint accuracy: the percentage of correctly identified distortion type and severity level among all parseable responses. Calculated as (number of correct joint predictions / number of answered prompts) * 100.

## Input / output format

**Input**: A single distorted image accompanied by a prompt requesting identification of the distortion type and its severity level.

**Output**: A JSON object containing the predicted distortion type and severity level.

## Scoring recipe

```python
def compute_acc(predictions, gold):
    answered = [(p, g) for p, g in zip(predictions, gold) if p is not None]
    if not answered:
        return 0.0
    correct = sum(1 for p, g in answered if p['type'] == g['type'] and p['level'] == g['level'])
    return (correct / len(answered)) * 100
```

## Common pitfalls

- Thinking/CoT variants frequently produce unparseable responses (up to 3.4%) when they exhaust the token budget on reasoning steps; these are excluded from accuracy calculations, artificially inflating performance on a slightly easier effective subset.
- Total parameter count is a misleading metric for MoE models on this task; performance correlates better with active parameters and visual encoder quality rather than raw scale.
- Severity discrimination is the dominant failure mode for both models and humans, often causing same-type/wrong-level confusions even when distortion type is correctly identified.

## Evidence (verbatim from paper)

> Table 3: Main results. Answered-only joint accuracy (Acc.) with bootstrap 95% confidence intervals, type-balanced accuracy (T-Bal.), level-balanced accuracy (L-Bal.), and unparseable-response rate (Unp.) over the full 13,500-prompt evaluation.

## Citation

```bibtex
@misc{goyal2026distortbench,
  title={DistortBench: Benchmarking Vision Language Models on Image Distortion Identification},
  author={Goyal et al. (2026)},
  year={2026},
  note={arXiv:2604.19966}
}
```

- arXiv: 2604.19966