Open Finllm Leaderboard Eval

---
name: open-finllm-leaderboard-eval
description: Evaluates the capability of financial LLMs and agents across seven core financial task categories, including information extraction, sentiment analysis, question answering, text generation, risk management, forecasting, and decision-making. The benchmark aggregates 42 existing financial datasets to provide a standardized comparison of model performance and compliance readiness. Use when the user wants to benchmark on Open FinLLM Leaderboard (42 financial datasets), or asks about evaluating this task. Reports average score across all tasks.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.19073
  bibtex_key: lin2026finllmleaderboard
  confidence: medium
---

# open-finllm-leaderboard-eval

> Evaluation and Benchmarking Suite for Financial Large Language Models and Agents — Lin et al. (2026) (arXiv:2602.19073, 2026)

## What this evaluates

Evaluates the capability of financial LLMs and agents across seven core financial task categories, including information extraction, sentiment analysis, question answering, text generation, risk management, forecasting, and decision-making. The benchmark aggregates 42 existing financial datasets to provide a standardized comparison of model performance and compliance readiness.

## Datasets

- **Open FinLLM Leaderboard (42 financial datasets)** — total ?; splits: test (-1)

## Metrics

- `average score across all tasks` **(primary)** — range: other
  - Not explicitly defined in the provided text; described as an aggregate of per-task scores computed across the 42 datasets organized into seven categories.
- `Model Openness Framework (MOF) class` — range: other
  - A compliance rating (Class I, II, or III) that evaluates model licensing and training data transparency to mitigate open-washing risks.

## Input / output format

**Input**: Multimodal financial data and prompts corresponding to the 42 datasets across seven task categories (IE, TA, QA, TG, RM, FO, DM).

**Output**: Task-specific predictions or generated text, plus an MOF openness class rating (Class I, II, or III).

## Scoring recipe

```python
scores = []
for dataset in datasets_42:
    pred = model.predict(dataset)
    scores.append(compute_task_metric(pred, dataset.gold))
overall_score = mean(scores)
openness_class = evaluate_openness(model_license, training_data)
return overall_score, openness_class
```

## Common pitfalls

- The benchmark aggregates 42 pre-existing datasets rather than defining a single unified test set, so results are highly task-dependent and not directly comparable across categories.
- The 'openness class' metric evaluates licensing and data transparency compliance, not model capability, and is scored separately from task accuracy.
- Exact evaluation formulas, dataset splits, and versioning are not provided in this section; they are hosted externally on the Hugging Face space and documentation site.

## Evidence (verbatim from paper)

> The current $42$ financial datasets are organized into seven categories, as given in Table [1]. The leaderboard on Huggingface is designed to show transparent and comprehensive evaluation results for financial LLMs. Leaderboard Table. The central part is a table containing: rank, model name, average score across all tasks, openness class (Class I, II, III under MOF in Section [5.3])

## Citation

```bibtex
@misc{lin2026finllmleaderboard,
  title={Evaluation and Benchmarking Suite for Financial Large Language Models and Agents},
  author={Lin et al. (2026)},
  year={2026},
  note={arXiv:2602.19073}
}
```

- arXiv: 2602.19073