Bizfinbench Eval

---
name: bizfinbench-eval
description: Evaluates LLMs on real-world financial reasoning tasks, including numerical calculation, temporal reasoning, information extraction, prediction recognition, and knowledge-based QA in Chinese. It probes the models' ability to handle noisy, context-dependent financial data and produce structured, reasoned outputs. Use when the user wants to benchmark on BizFinBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19457
  bibtex_key: lu2025bizfinbench
  confidence: high
---

# bizfinbench-eval

> BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs — Lu et al. (2025) (arXiv:2505.19457, 2025)

## What this evaluates

Evaluates LLMs on real-world financial reasoning tasks, including numerical calculation, temporal reasoning, information extraction, prediction recognition, and knowledge-based QA in Chinese. It probes the models' ability to handle noisy, context-dependent financial data and produce structured, reasoned outputs.

## Datasets

- **BizFinBench** — total 6781; splits: test (6781); repo https://github.com/HiThink-Research/BizFinBench

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered queries per task, calculated as (correct predictions / total predictions) * 100. Scores are reported per task and averaged across all nine tasks.

## Input / output format

**Input**: Chinese-language financial queries/tasks spanning nine categories: AEA, FNC, FTR, FTU, FQA, FDD, ER, SP, and FNER.

**Output**: Strictly formatted JSON containing two mandatory fields: 'cot' (detailed chain-of-thought logic trace) and 'Answer' (final conclusion).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        model_answer = json.loads(pred)['Answer']
        if model_answer.strip().lower() == gold_ans.strip().lower():
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Models must output strictly valid JSON with exact field names ('cot' and 'Answer'); parsing failures are common if formatting constraints are ignored.
- Evaluation uses GPT-4o as a unified judge, which may introduce bias or inconsistency; the paper introduces IteraJudge to mitigate this, but standard evaluation relies on the judge's correlation with human/expert labels.
- Tasks are in Chinese and require domain-specific financial knowledge; models trained primarily on English data may underperform significantly.

## Evidence (verbatim from paper)

> All LLMs were configured with a maximum generation length of 1,024 tokens, temperature parameter T=0, and batch size B=1000. We employed GPT-4o as the unified evaluation judge. ... we constrained all models to produce strictly JSON-formatted responses containing two mandatory fields: ① Chain-of-Thought(cot): Detailed logic trace with intermediate steps; and ② Answer: Final conclusion derived after reasoning.

## Citation

```bibtex
@misc{lu2025bizfinbench,
  title={BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs},
  author={Lu et al. (2025)},
  year={2025},
  note={arXiv:2505.19457}
}
```

- arXiv: 2505.19457