Probenc Eval

---
name: probenc-eval
description: Evaluates multimodal foundation models on open-ended, expert-level queries across 10 professional domains, probing visual perception, domain knowledge, and long-context reasoning in single-round, multi-lingual, and multi-turn settings. Use when the user wants to benchmark on ProBench, or asks about evaluating this task. Reports ELO rating.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.06885
  bibtex_key: yang2025probenc
  confidence: high
---

# probenc-eval

> ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks — Yan Yang et al. (2025) (arXiv:2503.06885, 2025)

## What this evaluates

Evaluates multimodal foundation models on open-ended, expert-level queries across 10 professional domains, probing visual perception, domain knowledge, and long-context reasoning in single-round, multi-lingual, and multi-turn settings.

## Datasets

- **ProBench** — total 4000; splits: test (4000)

## Metrics

- `ELO rating` **(primary)** — range: other
  - Pairwise comparison score derived from win rates against a baseline model, de-biased by swapping presentation order. Higher values indicate stronger relative performance.
- `Win Rate (WR)` — range: percent
  - Percentage of pairwise comparisons won by a model against the baseline or other evaluated models.

## Input / output format

**Input**: Multimodal prompts (images and text) representing open-ended expert queries from professional workflows.

**Output**: Open-ended text responses generated by the MLLM.

## Scoring recipe

```python
def compute_metrics(predictions, queries, judge='gpt-4o-2024-08-06', baseline='gpt-4o-2024-05-13'):
    wins = {m: 0 for m in predictions}
    for q in queries:
        for m1, m2 in pairwise(predictions):
            # Swap order twice to de-bias
            judge_out = judge.compare(m1.response, m2.response, q)
            wins[judge_out.winner] += 1
    elo = calculate_elo_debiased(wins, baseline)
    wr = {m: wins[m]/total for m in wins}
    return elo, wr
```

## Common pitfalls

- Relies on an LLM-as-a-Judge pipeline rather than ground-truth labels, making results sensitive to judge model selection and prompt design.
- ELO ratings are relative to a fixed baseline (gpt-4o-2024-05-13); changing the baseline or judge shifts relative rankings.
- Open-ended outputs prevent exact-match scoring, requiring careful aggregation of pairwise wins to compute final metrics.

## Evidence (verbatim from paper)

> Our MLLM judge utilizes gpt-4o-2024-08-06 with greedy sampling for consistent and reproducible evaluation. For pairwise comparisons in Elo rating calculations, we set gpt-4o-2024-05-13 as the baseline, evaluate each model twice by swapping the presentation order for each user query, and de-bias the ELO ratings by following the methodology of (Li et al., 2024c).

## Citation

```bibtex
@misc{yang2025probenc,
  title={ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks},
  author={Yan Yang et al. (2025)},
  year={2025},
  note={arXiv:2503.06885}
}
```

- arXiv: 2503.06885