Cfbenchmark Mm Eval

---
name: cfbenchmark-mm-eval
description: Evaluates multimodal large language models' ability to interpret financial charts, tables, and diagrams in Chinese, and answer domain-specific questions. It probes visual reasoning, statistical and structural analysis, and financial concept comprehension under zero-shot conditions. Use when the user wants to benchmark on CFBenchmark-MM, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.13055
  bibtex_key: li2025cfbenchmarkmm
  confidence: high
---

# cfbenchmark-mm-eval

> CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model — Jiangtong Li et al. (2025) (arXiv:2506.13055, 2025)

## What this evaluates

Evaluates multimodal large language models' ability to interpret financial charts, tables, and diagrams in Chinese, and answer domain-specific questions. It probes visual reasoning, statistical and structural analysis, and financial concept comprehension under zero-shot conditions.

## Datasets

- **CFBenchmark-MM** — total 9356; splits: test (9356)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered objective (multiple-choice) questions. Calculated as the number of exact-match correct predictions divided by the total number of objective questions.
- `subjective_score` — range: percent
  - Percentage score awarded to short-answer responses based on point-by-point correctness against reference answers. Evaluated via human or LLM judgment on a 0-100 scale.

## Input / output format

**Input**: A Chinese question {question}, optionally accompanied by a financial chart/image <IMAGE> and a textual caption {caption}. The prompt instructs the model to examine chart components (axes, legends, data points) and answer either as a multiple-choice selection or a point-by-point short answer.

**Output**: For multiple-choice questions: a list of all correct option letters. For short-answer questions: a structured, point-by-point textual response addressing each part of the query.

## Scoring recipe

```python
def calc_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if set(p) == set(g))
    return correct / len(golds) * 100

def calc_subjective_score(predictions, golds):
    # Score based on point-by-point alignment with reference answers
    # Typically 0-100 scale as reported in paper
    scores = [score_response(p, g) for p, g in zip(predictions, golds)]
    return sum(scores) / len(scores)
```

## Common pitfalls

- Models frequently misinterpret visual elements like axes, legends, or data points, leading to incorrect numerical extraction.
- Adding generated captions to the input can degrade performance for some models (e.g., GPT-4V) due to redundant or conflicting information.
- Financial concept misunderstandings (e.g., confusing compound vs. average growth rates) persist even when models correctly read the chart.
- Evaluation excludes models lacking Chinese language support, which may bias open-source model comparisons.

## Evidence (verbatim from paper)

> However, even advanced MLLMs like GPT-4V achieve only 52% accuracy and 38% score in objective and subjective questions, falling short of real-world application requirements.

## Citation

```bibtex
@misc{li2025cfbenchmarkmm,
  title={CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model},
  author={Jiangtong Li et al. (2025)},
  year={2025},
  note={arXiv:2506.13055}
}
```

- arXiv: 2506.13055