Mme Finance Eval

---
name: mme-finance-eval
description: Evaluates multimodal large language models' ability to understand and reason over financial charts, tables, and documents. It probes fine-grained visual perception, spatial reasoning, numerical calculation, and complex financial decision-making in a domain-specific context. Use when the user wants to benchmark on MME-Finance, or asks about evaluating this task. Reports LLM-based score (0-5).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.03314
  bibtex_key: gan2024mmefinance
  confidence: high
---

# mme-finance-eval

> MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning — Gan et al. (2024) (arXiv:2411.03314, 2024)

## What this evaluates

Evaluates multimodal large language models' ability to understand and reason over financial charts, tables, and documents. It probes fine-grained visual perception, spatial reasoning, numerical calculation, and complex financial decision-making in a domain-specific context.

## Datasets

- **MME-Finance** — total 2274; splits: test (-1)

## Metrics

- `LLM-based score (0-5)` **(primary)** — range: percent
  - An LLM evaluator assigns a score from 0 to 5 per sample. This score is multiplied by 20% to be normalized to a 0-100 scale.

## Input / output format

**Input**: Image, question, ground truth answer, and the model's generated response.

**Output**: A normalized score from 0 to 100 (derived from an LLM judge's 0-5 rating).

## Scoring recipe

```python
def compute_metric(predictions, gold):
    scores = []
    for img, q, gt, pred in zip(predictions['images'], predictions['questions'], gold['ground_truth'], predictions['responses']):
        prompt = f'Image: {img}, Question: {q}, Ground Truth: {gt}, Model Response: {pred}'
        raw_score = llm_evaluator.generate(prompt) # returns 0-5
        scores.append(raw_score * 20)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Relies on an LLM-as-a-judge, which may introduce bias or inconsistency compared to human experts.
- The 'Not Applicable' (NA) task explicitly prompts models to opt-out, potentially lowering the measured hallucination rate compared to standard open-ended evaluation.
- Performance on mobile photographs is heavily influenced by image resolution and capture angle, conflating visual quality with reasoning capability.

## Evidence (verbatim from paper)

> We fill the prompt template with image, question, ground truth, and response from an MLLM, and take the filled prompt into an LLM-based evaluator for generating a score range from 0 to 5 for one sample. The scores are multiplied by 20% to be normalized.

## Citation

```bibtex
@misc{gan2024mmefinance,
  title={MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning},
  author={Gan et al. (2024)},
  year={2024},
  note={arXiv:2411.03314}
}
```

- arXiv: 2411.03314