Iris Benchmark Eval

---
name: iris-benchmark-eval
description: Probes fairness across understanding and generation tasks in Unified Multimodal Large Language Models (UMLLMs) by measuring Ideal Fairness, Real-world Fidelity, and Bias Inertia & Steerability across demographic attributes. It reveals systemic trade-offs, generation gaps, and personality splits that single-task or single-metric evaluations miss. Use when the user wants to benchmark on IRIS Benchmark, or asks about evaluating this task. Reports IRIS-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.00590
  bibtex_key: zhao2026irisbenchmark
  confidence: high
---

# iris-benchmark-eval

> Fair in Mind, Fair in Action? A Synchronous Benchmark for Understanding and Generation in UMLLMs — Zhao et al. (2026) (arXiv:2603.00590, 2026)

## What this evaluates

Probes fairness across understanding and generation tasks in Unified Multimodal Large Language Models (UMLLMs) by measuring Ideal Fairness, Real-world Fidelity, and Bias Inertia & Steerability across demographic attributes. It reveals systemic trade-offs, generation gaps, and personality splits that single-task or single-metric evaluations miss.

## Datasets

- **IRIS Benchmark** — total ?; splits: test (-1)

## Metrics

- `IRIS-Score` **(primary)** — range: other
  - Aggregated score across three dimensions (Ideal Fairness, Real-world Fidelity, Bias Inertia & Steerability) for both understanding and generation tasks. Higher values indicate better fairness performance.
- `Ideal Fairness (IFS)` — range: other
  - Measures alignment with normative fairness principles across demographic attributes (age, gender, skin tone) in model outputs.
- `Real-world Fidelity (RFS)` — range: other
  - Assesses how accurately and fairly models reflect real-world demographic distributions and contexts.
- `Bias Inertia & Steerability (BIS)` — range: other
  - Evaluates the model's willingness and ability to adjust outputs when prompted to reduce bias or change demographic representation.

## Input / output format

**Input**: Multimodal prompts (text and/or images) requiring the model to generate or interpret content involving demographic attributes (age, gender, skin tone) for fairness assessment.

**Output**: Generated text and/or images, which are subsequently analyzed by the ARES classifier and scored across the three fairness dimensions.

## Scoring recipe

```python
def compute_iris_scores(predictions, gold_demographics):
    # 1. Use ARES classifier to detect age, gender, skin tone in outputs
    detected = ares_classifier.predict(predictions)
    # 2. Calculate dimension scores based on fairness criteria
    ifs = calculate_ideal_fairness(detected, gold_demographics)
    rfs = calculate_real_world_fidelity(detected, gold_demographics)
    bis = calculate_bias_inertia_steerability(detected, gold_demographics)
    # 3. Aggregate into overall IRIS-Score
    iris_score = aggregate_scores(ifs, rfs, bis)
    return {'IFS': ifs, 'RFS': rfs, 'BIS': bis, 'IRIS-Score': iris_score}
```

## Common pitfalls

- Assuming a single optimal model exists; the benchmark explicitly demonstrates inherent trade-offs across fairness dimensions.
- Evaluating understanding and generation tasks in isolation; the synchronous dual-task design is required to detect phenomena like the 'generation gap' and 'personality splits'.
- Treating Bias Inertia & Steerability (BIS) as a monolithic metric; validation shows it comprises distinct 'willingness' and 'ability' components.

## Evidence (verbatim from paper)

> All scores are scaled such that higher values indicate better performance (↑). For each metric, the best performance (highest score) is marked with † and the worst (lowest) with ‡. The left panel details the fairness scores across Understanding (Und) and Generation (Gen) tasks. The right panel displays the overall IRIS-Score and the diagnosed model personality profiles.

## Citation

```bibtex
@misc{zhao2026irisbenchmark,
  title={Fair in Mind, Fair in Action? A Synchronous Benchmark for Understanding and Generation in UMLLMs},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2603.00590}
}
```

- arXiv: 2603.00590