Worldsense Eval

---
name: worldsense-eval
description: Evaluates multimodal large language models' ability to perform real-world omni-modal understanding by jointly processing tightly coupled audio and video inputs. It probes complex temporal reasoning, cross-modal integration, and fine-grained perception across diverse everyday scenarios. Use when the user wants to benchmark on WorldSense, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.04326
  bibtex_key: hong2025worldsense
  confidence: high
---

# worldsense-eval

> WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs — Jack Hong et al. (arXiv:2502.04326, 2025)

## What this evaluates

Evaluates multimodal large language models' ability to perform real-world omni-modal understanding by jointly processing tightly coupled audio and video inputs. It probes complex temporal reasoning, cross-modal integration, and fine-grained perception across diverse everyday scenarios.

## Datasets

- **WorldSense** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the number of exact matches between model predictions and ground-truth answers divided by the total number of questions, multiplied by 100 to yield a percentage.

## Input / output format

**Input**: Video frames (extracted per model-specific configurations) paired with the corresponding audio track. Ablation variants also accept video with transcribed subtitles/captions.

**Output**: Natural language text answer to a question about the video content.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if p.strip().lower() == g.strip().lower())
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Evaluating on unimodal inputs (video-only or audio-only) significantly underestimates performance, as accuracy drops by ~15% when either modality is removed.
- Using transcribed subtitles instead of raw audio fails to capture prosody and paralinguistic cues, leading to misleading metrics for true omni-modal integration.
- Direct string matching is used without explicit mention of LLM-as-a-judge or fuzzy matching, so paraphrased but correct answers may be incorrectly penalized.

## Evidence (verbatim from paper)

> Gemini 1.5 Pro, capable of processing both audio and visual information, achieves the highest accuracy of 48.0%.

## Citation

```bibtex
@misc{hong2025worldsense,
  title={WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs},
  author={Jack Hong et al.},
  year={2025},
  note={arXiv:2502.04326}
}
```

- arXiv: 2502.04326