Unipercept Bench Eval

---
name: unipercept-bench-eval
description: Evaluates multimodal large language models on perceptual-level image understanding across three domains: Image Aesthetics & Art (IAA), Image Quality Assessment (IQA), and Image Structure & Texture Assessment (ISTA). It probes both continuous visual rating (VR) and discrete visual question answering (VQA) capabilities. Use when the user wants to benchmark on UniPercept-Bench, or asks about evaluating this task. Reports Acc..
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.21675
  bibtex_key: cao2025unipercept
  confidence: high
---

# unipercept-bench-eval

> UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texture — Cao et al. (2025) (arXiv:2512.21675, 2025)

## What this evaluates

Evaluates multimodal large language models on perceptual-level image understanding across three domains: Image Aesthetics & Art (IAA), Image Quality Assessment (IQA), and Image Structure & Texture Assessment (ISTA). It probes both continuous visual rating (VR) and discrete visual question answering (VQA) capabilities.

## Datasets

- **UniPercept-Bench** — total ?; splits: test (-1); repo https://github.com/thunderbolt215/UniPercept

## Metrics

- `Acc.` **(primary)** — range: percent
  - Percentage of correctly predicted answers for VQA tasks, averaged across all question templates and domains.
- `(SRCC+PLCC)/2` — range: [0, 1]
  - Average of Spearman Rank Correlation Coefficient (SRCC) and Pearson Linear Correlation Coefficient (PLCC) between predicted continuous scores and ground-truth ratings for VR tasks.

## Input / output format

**Input**: An image paired with a text prompt. For VQA, the prompt is a specific question (e.g., Level Prediction, How, What, Which, Why, Yes-No). For VR, the prompt is a task-specific instruction to elicit a quantitative score.

**Output**: For VQA: a categorical answer string matching the ground-truth options. For VR: a continuous numerical score.

## Scoring recipe

```python
# VQA Accuracy
vqa_acc = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)

# Visual Rating Score
srcc = spearmanr(predictions, gold)[0]
plcc = pearsonr(predictions, gold)[0]
vr_score = (srcc + plcc) / 2
```

## Common pitfalls

- Visual Rating tasks require generating continuous numerical scores, which often causes general MLLMs to produce hallucinated or unstable predictions compared to specialized models.
- Specialized models used for comparison are trained exclusively on in-domain datasets, so cross-domain evaluation may require retraining or careful prompt engineering to ensure fair comparison.
- VQA accuracy varies significantly by question template (e.g., Level Prediction is much harder than Yes-No), so reporting only the overall average can mask domain-specific weaknesses.

## Evidence (verbatim from paper)

> For VQA, all models were provided with identical prompts corresponding to each question, and their generated answers were compared against the ground-truth options. For VR, we designed task-specific prompts for models lacking a dedicated scoring interface to elicit quantitative predictions... Figure 7: Results on UniPercept-Bench.VQA and VR are evaluated by Acc. and $(SRCC+PLCC)/2$.

## Citation

```bibtex
@misc{cao2025unipercept,
  title={UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texture},
  author={Cao et al. (2025)},
  year={2025},
  note={arXiv:2512.21675}
}
```

- arXiv: 2512.21675