Mixeval X Eval

---
name: mixeval-x-eval
description: Evaluates multi-modal and agent models across any-to-any generation and action-planning tasks using real-world data mixtures. It probes capabilities in vision-language understanding, audio-language understanding, text-to-media generation, and API-level action planning. Use when the user wants to benchmark on MixEval-X, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.13754
  bibtex_key: ni2024mixevalx
  confidence: high
---

# mixeval-x-eval

> MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures — Ni et al. (2024) (arXiv:2410.13754, 2024)

## What this evaluates

Evaluates multi-modal and agent models across any-to-any generation and action-planning tasks using real-world data mixtures. It probes capabilities in vision-language understanding, audio-language understanding, text-to-media generation, and API-level action planning.

## Datasets

- **MixEval-X** — total ?; splits: general (-1), hard (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly generated responses for MMU tasks (Image2Text, Video2Text, Audio2Text).
- `Elo score` — range: other
  - Paired comparison scores derived from crowd-sourced human pairwise rankings using the Bradley-Terry model for MMG tasks (Text2Image, Text2Video, Text2Audio).

## Input / output format

**Input**: Varies by task: MMU tasks receive image/video/audio plus text prompts; MMG tasks receive text prompts; Agent tasks receive text/image plus text prompts describing the environment and user goal.

**Output**: Varies by task: MMU tasks require text responses; MMG tasks require generated image/video/audio; Agent tasks require sequences of API-level actions.

## Scoring recipe

```python
if task in MMU_TASKS:
    score = 1.0 if prediction == gold else 0.0
    metric = mean(score)
elif task in MMG_TASKS:
    # Human pairwise ranking via MTurk
    # Bradley-Terry model converts pairwise wins/losses to Elo scores
    metric = compute_elo_score(pairwise_human_ratings)
elif task in AGENT_TASKS:
    # Task success based on API execution or action sequence validity
    metric = task_success_rate(predictions, environment_state)
```

## Common pitfalls

- MMG tasks rely on human pairwise rankings rather than automatic metrics, which fail to capture output quality nuances.
- Input formatting must be standardized; MMG tasks require caption-like prompts rewritten by GPT-4, otherwise models may fail.
- For Video2Text, frame count limits significantly impact performance, especially on long videos.

## Evidence (verbatim from paper)

> Gemini 1.5 Pro ranking first at $62.7\%$ accuracy on the general split but showing room for improvement on the Audio2Text-Hard split. ... The results are derived using the Bradley-Terry model, based on crowd-sourced user preferences.

## Citation

```bibtex
@misc{ni2024mixevalx,
  title={MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures},
  author={Ni et al. (2024)},
  year={2024},
  note={arXiv:2410.13754}
}
```

- arXiv: 2410.13754