Financial Exam Qa Eval

---
name: financial-exam-qa-eval
description: Evaluates whether LLMs can perform domain-level conceptual understanding and precise financial reasoning across multilingual professional certification exams. It probes analytical rigor and regulatory knowledge integration rather than simple factual recall. Use when the user wants to benchmark on EFPA, GRFinQA, CFA, CPA, BBF, SAHM, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.10886
  bibtex_key: xie2026finmmeval
  confidence: high
---

# financial-exam-qa-eval

> The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systems — Xie et al. (2026) (arXiv:2602.10886, 2026)

## What this evaluates

Evaluates whether LLMs can perform domain-level conceptual understanding and precise financial reasoning across multilingual professional certification exams. It probes analytical rigor and regulatory knowledge integration rather than simple factual recall.

## Datasets

- **EFPA** — total 230; splits: test (-1)
- **GRFinQA** — total 268; splits: test (-1)
- **CFA** — total 600; splits: test (-1)
- **CPA** — total 300; splits: test (-1)
- **BBF** — total ?; splits: test (-1)
- **SAHM** — total 873; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Accuracy is defined as the proportion of correctly identified options in the test set, calculated as the number of correct predictions divided by the total number of questions.

## Input / output format

**Input**: A stand-alone multiple-choice question Q with four candidate answers {A1, A2, A3, A4} in one of six languages (English, Chinese, Arabic, Hindi, Greek, Spanish).

**Output**: The model must select and output the single correct answer A* from the four options.

## Scoring recipe

```python
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = correct / total
```

## Common pitfalls

- Models may rely on surface pattern recognition or factual recall instead of the required analytical rigor and conceptual reasoning.
- Language coverage is treated as a configurable dimension rather than defining separate subtasks, so results across different language subsets may not be directly comparable.

## Evidence (verbatim from paper)

> The models are required to output the correct answer, and the performance is measured in terms of accuracy, defined as the proportion of correctly identified options in the test set.

## Citation

```bibtex
@misc{xie2026finmmeval,
  title={The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systems},
  author={Xie et al. (2026)},
  year={2026},
  note={arXiv:2602.10886}
}
```

- arXiv: 2602.10886