Dianjin R1 Eval

---
name: dianjin-r1-eval
description: Evaluates large language models' financial reasoning capabilities and general problem-solving skills across multiple benchmarks. It measures how well models can answer domain-specific financial questions and general math/science reasoning tasks, while also assessing compliance rule adherence in Chinese financial contexts. Use when the user wants to benchmark on CFLUE, FinQA, CCC, MATH-500, GPQA-Diamond, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.15716
  bibtex_key: [REDACTED]
  confidence: high
---

# dianjin-r1-eval

> DianJin-R1: Evaluating and Enhancing Financial Reasoning in Large Language Models — Jie Zhu et al. (2025) (arXiv:2504.15716, 2025)

## What this evaluates

Evaluates large language models' financial reasoning capabilities and general problem-solving skills across multiple benchmarks. It measures how well models can answer domain-specific financial questions and general math/science reasoning tasks, while also assessing compliance rule adherence in Chinese financial contexts.

## Datasets

- **CFLUE** — total 3864; splits: test (3864)
- **FinQA** — total 1147; splits: test (1147)
- **CCC** — total 200; splits: test (200)
- **MATH-500** — total 500; splits: test (500)
- **GPQA-Diamond** — total 198; splits: test (198)

## Metrics

- `accuracy` **(primary)** — range: percent
  - The proportion of correctly answered questions out of the total number of test instances. Average accuracy is computed across all five test sets.

## Input / output format

**Input**: Natural language financial reasoning questions or general math/science problems, provided in either Chinese or English.

**Output**: Predicted answer string. For multiple-choice questions, the predicted option; for open-ended questions, the extracted final answer or compliance judgment.

## Scoring recipe

```python
def compute_accuracy(predictions, golds, dataset_name):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if dataset_name in ['FinQA', 'CCC']:
            is_correct = gpt4o_evaluate(pred, gold, prompt_file)
        else:
            extracted_pred = extract_answer(pred)
            is_correct = (extracted_pred == gold)
        correct += is_correct
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Evaluation uses different methods per dataset: rule-based exact matching for CFLUE/MATH/GPQA, but GPT-4o LLM-as-a-judge for FinQA and CCC, which can introduce inconsistency.
- CCC is a proprietary in-house dataset and not publicly available, limiting direct replication of results.
- RL training data was exclusively Chinese (from CFLUE), which caused a performance drop on the English FinQA benchmark due to language mismatch.

## Evidence (verbatim from paper)

> For each dataset, we report the accuracy—defined as the proportion of correctly answered questions—and compute the average accuracy across all test sets. Among them, CFLUE and CCC are Chinese-language datasets, while the others are in English. The detailed statistics of these test sets are summarized in Table 2. For FinQA and CCC, we use GPT-4o to evaluate the correctness of each answer, following the prompts shown in Figure 8 and Figure 9 in Appendix B. For the other test sets, we extract the predicted answers using rule-based methods and compare them directly with the gold answers.

## Citation

```bibtex
@misc{zhu2025dianjinr1,
  title={DianJin-R1: Evaluating and Enhancing Financial Reasoning in Large Language Models},
  author={Jie Zhu et al. (2025)},
  year={2025},
  note={arXiv:2504.15716}
}
```

- arXiv: 2504.15716