Finmme Eval

---
name: finmme-eval
description: Evaluates financial multi-modal reasoning capabilities of models on chart-based analysis and domain-specific knowledge. It probes perception, analysis, and reasoning across 18 financial domains and 6 asset classes using multiple-choice and computational problems. Use when the user wants to benchmark on FinMME, or asks about evaluating this task. Reports FinScore.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.24714
  bibtex_key: luo2025finmme
  confidence: high
---

# finmme-eval

> FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation — Luo et al. (2025) (arXiv:2505.24714, 2025)

## What this evaluates

Evaluates financial multi-modal reasoning capabilities of models on chart-based analysis and domain-specific knowledge. It probes perception, analysis, and reasoning across 18 financial domains and 6 asset classes using multiple-choice and computational problems.

## Datasets

- **FinMME** — total 11000; splits: test (-1); repo https://github.com/luo-junyu/FinMME

## Metrics

- `FinScore` **(primary)** — range: percent
  - FinScore = F * (1 - P_H), where F is the domain-normalized average raw score and P_H is the hallucination penalty rate. Raw score per MCQ is max(0, c/n - i/s), with c=correct selections, n=total options, i=incorrect selections, s=total selections. F averages raw scores within each of the 18 domains, then averages across domains. P_H is the mean ratio of incorrect to total selections across all questions.

## Input / output format

**Input**: Financial charts/images paired with multiple-choice questions (single and multiple answer) and computational problems.

**Output**: Selected options for MCQs and computed answers for problems.

## Scoring recipe

```python
def compute_fin_score(predictions, gold, domains):
    domain_scores = {}
    for k in domains:
        q_scores = []
        for q in domain_questions[k]:
            c = count_correct(predictions[q], gold[q])
            n = total_options(q)
            i = count_incorrect(predictions[q], gold[q])
            s = total_selected(predictions[q])
            raw = max(0, c/n - i/s)
            q_scores.append(raw)
        domain_scores[k] = mean(q_scores)
    F = mean(domain_scores.values())
    P_H = mean([count_incorrect(p, g)/total_selected(p) for p, g in zip(predictions, gold)])
    return F * (1 - P_H)
```

## Common pitfalls

- Multiple-answer questions use a non-standard scoring formula that penalizes over-selection, so standard exact-match or recall metrics will misrepresent performance.
- Domain normalization averages scores within each of the 18 domains first, then averages across domains, preventing domains with more questions from dominating the final score.
- FinScore is multiplicative, meaning a model with high accuracy but high hallucination rate will receive a drastically lower score than one with slightly lower accuracy but higher reliability.

## Evidence (verbatim from paper)

> We first define the hallucination penalty rate $P_{H}$, which represents the average ratio of incorrect selections across the dataset: $P_{H}=\text{mean}\left(\frac{i}{s}\right)$. The final FinScore combines the domain-normalized score with the hallucination penalty: $\mathcal{F}=F\cdot(1-P_{H})$, where $F$ is the domain-normalized average score across all questions and $P_{H}$ is the hallucination penalty rate.

## Citation

```bibtex
@misc{luo2025finmme,
  title={FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2505.24714}
}
```

- arXiv: 2505.24714