Japanese Financial Bench Eval

---
name: japanese-financial-bench-eval
description: Evaluates large language models on Japanese financial domain knowledge across five distinct tasks: sentiment analysis, fundamental financial knowledge, CPA auditing, and two levels of financial planner exam questions. It probes the models' ability to understand and reason over domain-specific multiple-choice questions in Japanese. Use when the user wants to benchmark on chabsa, cma Basics, cpa Audit, fp2, security_sales_1, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.15062
  bibtex_key: [REDACTED]
  confidence: high
---

# japanese-financial-bench-eval

> Construction of a Japanese Financial Benchmark for Large Language Models — Hirano et al. (2024) (arXiv:2403.15062, 2024)

## What this evaluates

Evaluates large language models on Japanese financial domain knowledge across five distinct tasks: sentiment analysis, fundamental financial knowledge, CPA auditing, and two levels of financial planner exam questions. It probes the models' ability to understand and reason over domain-specific multiple-choice questions in Japanese.

## Datasets

- **chabsa** — total ?; splits: test (-1); repo https://github.com/pfnet-research/japanese-lm-fin-harness
- **cma Basics** — total ?; splits: test (-1); repo https://github.com/pfnet-research/japanese-lm-fin-harness
- **cpa Audit** — total ?; splits: test (-1); repo https://github.com/pfnet-research/japanese-lm-fin-harness
- **fp2** — total ?; splits: test (-1); repo https://github.com/pfnet-research/japanese-lm-fin-harness
- **security_sales_1** — total ?; splits: test (-1); repo https://github.com/pfnet-research/japanese-lm-fin-harness

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted choices out of total instances. Calculated as (correct predictions / total instances) * 100.

## Input / output format

**Input**: Multiple-choice questions in Japanese, formatted with task-specific prompts and 0-4 shot examples.

**Output**: The model outputs the selected choice (determined by highest likelihood or earliest appearance in generation).

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Prompt tuning (0-4 shots) was performed per task, risking in-sample leakage.
- OpenAI API models were restricted to 0-shot due to cost, creating an unfair comparison.
- Content filters on OpenAI API blocked some responses, which were counted as incorrect.

## Evidence (verbatim from paper)

> To answer the multiple-choice questions, the likelihoods of the choices in the context were calculated and the choice with the highest likelihood was employed as the output. For GPT3.5 and GPT-4 series, the outputs with the temperature parameter set to 0 were obtained via API, and the choice that appeared earliest in the outputs was used as the output.

## Citation

```bibtex
@misc{hirano2024construction,
  title={Construction of a Japanese Financial Benchmark for Large Language Models},
  author={Hirano et al. (2024)},
  year={2024},
  note={arXiv:2403.15062}
}
```

- arXiv: 2403.15062