Finmr Eval

---
name: finmr-eval
description: Evaluates the multimodal financial reasoning capabilities of LLMs and MLLMs on expert-level question-answer pairs spanning 15 financial domains. It probes the models' ability to interpret complex visual data (charts, tables), apply domain-specific formulas, and perform multi-step logical calculations. Use when the user wants to benchmark on FinMR, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.06282
  bibtex_key: deng2025finmr
  confidence: high
---

# finmr-eval

> Understanding Financial Reasoning in AI: A Multimodal Benchmark and Error Learning Approach — Deng et al. (2025) (arXiv:2506.06282, 2025)

## What this evaluates

Evaluates the multimodal financial reasoning capabilities of LLMs and MLLMs on expert-level question-answer pairs spanning 15 financial domains. It probes the models' ability to interpret complex visual data (charts, tables), apply domain-specific formulas, and perform multi-step logical calculations.

## Datasets

- **FinMR** — total 3200; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Calculated as (number of correct predictions / total number of questions) * 100. Reported at the overall level, for sub-categories (Math, Expertise), and across 15 specific financial domains.

## Input / output format

**Input**: A question paired with an image (e.g., financial chart or table) or an image caption. For text-only LLMs, images are converted to captions using GPT-4o.

**Output**: A final answer, typically generated via Chain-of-Thought (CoT) or Error Feedback Learning (EFL) prompting. The model outputs reasoning steps followed by a definitive answer.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred.strip() == gold.strip():
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Confusing performance gains from GPT-4o-generated image captions with true multimodal reasoning, as captions artificially boost LLM scores without actual visual processing.
- Overlooking the 'answer not found' error type, where models fail to produce a final answer due to token repetition, which should be counted as incorrect.
- Failing to distinguish between mathematical reasoning and expertise reasoning tasks, as they require fundamentally different capabilities (logical rigor vs. contextual understanding).

## Evidence (verbatim from paper)

> In particular, the textual LLM DeepSeek-R1 and multimodal Gemini-1.5-Pro gained 71.88% and 82.06% overall accuracy, respectively. In contrast, open-source models such as LLaMa 3.2 and LLaVa-NEXT demonstrate significantly lower overall performances, with accuracies falling below 30%.

## Citation

```bibtex
@misc{deng2025finmr,
  title={Understanding Financial Reasoning in AI: A Multimodal Benchmark and Error Learning Approach},
  author={Deng et al. (2025)},
  year={2025},
  note={arXiv:2506.06282}
}
```

- arXiv: 2506.06282