Xmodbench Eval

---
name: xmodbench-eval
description: This benchmark probes the cross-modal consistency and reasoning capabilities of omni-language models by evaluating semantic equivalence across all six possible modality combinations (text, vision, audio) for both context and candidate inputs. It measures how well models maintain performance when modalities are swapped or combined, highlighting modality-specific biases and directional asymmetries. Use when the user wants to benchmark on XModBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.15148
  bibtex_key: wang2025xmodbench
  confidence: high
---

# xmodbench-eval

> XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models — Wang et al. (2025) (arXiv:2510.15148, 2025)

## What this evaluates

This benchmark probes the cross-modal consistency and reasoning capabilities of omni-language models by evaluating semantic equivalence across all six possible modality combinations (text, vision, audio) for both context and candidate inputs. It measures how well models maintain performance when modalities are swapped or combined, highlighting modality-specific biases and directional asymmetries.

## Datasets

- **XModBench** — total 60828; splits: test (60828)

## Metrics

- `accuracy` **(primary)** — range: [0, 100] percent
  - Percentage of correctly answered multiple-choice questions. Computed per modality configuration (e.g., Audio→Text, Vision→Text) and averaged across tasks and configurations to report overall performance and standard deviation.

## Input / output format

**Input**: Multiple-choice question where context and candidate options are provided in specific modality combinations (e.g., audio context with text candidates, text context with vision candidates).

**Output**: Selection of the correct candidate option from the provided set.

## Scoring recipe

```python
correct = 0
total = 0
for config in modality_configs:
    for q in dataset[config]:
        pred = model.predict(q.context, q.candidates)
        if pred == q.gold_answer:
            correct += 1
        total += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Confusing context vs. candidate modalities (e.g., treating A→T the same as T→A), which masks directional imbalance.
- Assuming modality substitution yields symmetric performance; the benchmark explicitly tests for asymmetries where swapping context/candidate roles changes accuracy significantly.
- Overlooking that audio-vision combinations without text anchors yield significantly lower scores, which is a key finding rather than a model failure.

## Evidence (verbatim from paper)

> Table 2 reports results across five task families and six cross-modal directions among Text, Vision, and Audio. The first subtable summarizes the average accuracy across all tasks for each modality configuration, while the remaining subtables present detailed performance within each task family.

## Citation

```bibtex
@misc{wang2025xmodbench,
  title={XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.15148}
}
```

- arXiv: 2510.15148