Opening Eval

---
name: opening-eval
description: Evaluates the quality of open-ended interleaved image-text generation by comparing model outputs against human-annotated references. It probes multimodal coherence, visual fidelity, and text-image alignment through pairwise battles and automated judge agreement. Use when the user wants to benchmark on OpenING, or asks about evaluating this task. Reports agreement.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.18499
  bibtex_key: zhou2024opening
  confidence: high
---

# opening-eval

> OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation — Zhou et al. (2024) (arXiv:2411.18499, 2024)

## What this evaluates

Evaluates the quality of open-ended interleaved image-text generation by comparing model outputs against human-annotated references. It probes multimodal coherence, visual fidelity, and text-image alignment through pairwise battles and automated judge agreement.

## Datasets

- **OpenING** — total 5400; splits: test (5400)

## Metrics

- `agreement` **(primary)** — range: percent
  - Measures the consistency between an automated judge and human evaluators. Calculated as the proportion of pairwise judgments where the judge's win/loss/tie decision matches the human's decision under a specified tie-handling strategy.
- `win rate` — range: percent
  - Indicates how often a model wins in pairwise comparisons. Computed using four tie-handling strategies: Force Dividing Tie (FDT), Without Tie (w/o Tie), With Tie counted as 0 (w/ Tie (0)), and With Tie counted as 0.5 (w/ Tie (.5)).

## Input / output format

**Input**: A task prompt requiring interleaved image-text generation, paired with a reference human-annotated output. Models generate outputs that are then presented in pairwise battles against the reference or other models.

**Output**: A pairwise judgment (Model A wins, Model B wins, or Tie) or a numerical score per criterion for the generated interleaved content.

## Scoring recipe

```python
def compute_win_rate(judgments, tie_strategy='FDT'):
    wins = 0
    total = 0
    for j in judgments:
        if tie_strategy == 'FDT':
            wins += 1 if j == 'A' else 0
            total += 1
        elif tie_strategy == 'w/o Tie':
            if j != 'Tie':
                wins += 1 if j == 'A' else 0
                total += 1
        elif tie_strategy == 'w/ Tie (0)':
            wins += 1 if j == 'A' else 0
            total += 1
        elif tie_strategy == 'w/ Tie (0.5)':
            wins += 0.5 if j == 'Tie' else (1 if j == 'A' else 0)
            total += 1
    return wins / total if total > 0 else 0

def compute_agreement(judge_judgments, human_judgments):
    matches = sum(1 for j, h in zip(judge_judgments, human_judgments) if j == h)
    return matches / len(human_judgments)
```

## Common pitfalls

- Win rates are highly sensitive to the tie-handling strategy; results using FDT, w/o Tie, and w/ Tie (0.5) are not directly comparable.
- Agreement scores depend heavily on the sampling size of battle pairs (4,320 pairs used); smaller samples yield unstable rankings.
- The 'seen' vs 'unseen' model split significantly impacts agreement, with unseen models showing lower agreement due to lack of training exposure.

## Evidence (verbatim from paper)

> Model performance are evaluated using two key metrics: win rate and agreement. Win rate indicates how often a model wins in pairwise comparisons. Four methods used to handle ties include 1) Force Dividing Tie (FDT): We force judges to assign ties with a more leaning model in rules and prompts, ensuring that every comparison round results in a decisive outcome. A win is attributed to A if a tie favors model A (Tie(A)), likewise for B. This metric allows for clear rankings without ambiguity. 2) Without Tie (w/o Tie): Tied comparisons are excluded; only matches with a clear winner are considered; 3) With Tie counted as 0 (w/ Tie (0)): Ties are included but do not contribute to the win count of either model; 4) With Tie counted as 0.5 (w/ Tie (.5)): Each tie contributes half a win to both models. Agreement measures the consistency between different evaluators (e.g., automated pipelines and human judgments) under the same tie-handling strategies.

## Citation

```bibtex
@misc{zhou2024opening,
  title={OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation},
  author={Zhou et al. (2024)},
  year={2024},
  note={arXiv:2411.18499}
}
```

- arXiv: 2411.18499