Flame Cer Eval

---
name: flame-cer-eval
description: Evaluates financial domain knowledge and certification exam readiness in Chinese and English. It probes models' ability to answer multiple-choice questions across 14 professional financial certifications with varying difficulty levels. Use when the user wants to benchmark on FLAME-Cer, or asks about evaluating this task. Reports accuracy rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.06211
  bibtex_key: guo2025flame
  confidence: high
---

# flame-cer-eval

> FLAME: Financial Large-Language Model Assessment and Metrics Evaluation — Guo et al. (2025) (arXiv:2501.06211, 2025)

## What this evaluates

Evaluates financial domain knowledge and certification exam readiness in Chinese and English. It probes models' ability to answer multiple-choice questions across 14 professional financial certifications with varying difficulty levels.

## Datasets

- **FLAME-Cer** — total 16000; splits: test (-1); repo https://github.com/FLAME-ruc/FLAME

## Metrics

- `accuracy rate` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total questions) * 100.

## Input / output format

**Input**: Multiple-choice question text in Chinese or English, followed by options (A, B, C, D, or E).

**Output**: A single letter corresponding to the selected option (e.g., 'A', 'B', 'C', 'D').

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = (correct / len(predictions)) * 100
return accuracy
```

## Common pitfalls

- Models may struggle with bilingual prompts (Chinese questions with English options or vice versa).
- Strict format adherence is required; outputting explanations alongside the letter may cause parsing failures.
- Financial calculations and regulatory details require precise domain knowledge; hallucination rates are high for complex actuarial or accounting questions.

## Evidence (verbatim from paper)

> The performance comparison of LLMs on the FLAME-Cer benchmark is shown in Table[1]. We could find that Baichuan4-Finance leads with an average accuracy rate of 93.62%, followed by Qwen2.5-72B-Instruct at 88.24%.

## Citation

```bibtex
@misc{guo2025flame,
  title={FLAME: Financial Large-Language Model Assessment and Metrics Evaluation},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2501.06211}
}
```

- arXiv: 2501.06211