Finset Eval

---
name: finset-eval
description: Evaluates financial LLMs across seven text-based tasks (sentiment analysis, NER, number understanding, summarization, stock movement prediction, credit scoring, firm disclosure) and three multimodal/hallucination tasks (ChartQA, FinVQA, FinTerms). It measures domain-specific reasoning, instruction following, and hallucination mitigation in financial contexts. Use when the user wants to benchmark on FinSet, ChartQA, FinVQA, FinTerms-MCQ, FinTerms-Gen, Finance Bench, or asks about evaluating this task. Reports Task Accuracy/F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.10986
  bibtex_key: bhatia2024fintral
  confidence: high
---

# finset-eval

> FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models — Bhatia et al. (2024) (arXiv:2402.10986, 2024)

## What this evaluates

Evaluates financial LLMs across seven text-based tasks (sentiment analysis, NER, number understanding, summarization, stock movement prediction, credit scoring, firm disclosure) and three multimodal/hallucination tasks (ChartQA, FinVQA, FinTerms). It measures domain-specific reasoning, instruction following, and hallucination mitigation in financial contexts.

## Datasets

- **FinSet** — total ?; splits: test (-1); repo https://github.com/UBC-NLP/fintral
- **ChartQA** — total ?; splits: test (-1)
- **FinVQA** — total ?; splits: test (-1)
- **FinTerms-MCQ** — total ?; splits: test (-1)
- **FinTerms-Gen** — total 150; splits: test (150)
- **Finance Bench** — total 150; splits: test (150)

## Metrics

- `Task Accuracy/F1` **(primary)** — range: [0, 1]
  - Reported as decimal scores (0.00–1.00) for SA, NER, NU, TS, SMP, CS, and FD. The exact metric (accuracy vs. F1) is not specified in the text.
- `Hallucination Index (HI)` — range: [0, 1]
  - Proportion of correctly generated definitions in the FinTerms-MCQ multiple-choice task. Calculated as correct predictions divided by total questions.
- `Human Evaluation` — range: other
  - Responses categorized into four quality tags (A: correct/satisfying, B: acceptable, C: significant errors, D: irrelevant) by two finance experts.
- `Finance Bench Accuracy` — range: percent
  - Percentage of correct, incorrect, and failed responses on open-book financial QA.

## Input / output format

**Input**: Text prompts for seven financial tasks (SA, NER, NU, TS, SMP, CS, FD); image+text prompts for ChartQA and FinVQA; multiple-choice questions for FinTerms-MCQ; open-book financial QA questions for Finance Bench.

**Output**: Task-specific outputs: labels/entities/numbers/summaries/movement directions/scores/disclosures; chart descriptions/answers; selected MCQ option; free-form QA answers.

## Scoring recipe

```python
# Text tasks (SA, NER, NU, TS, SMP, CS, FD)
pred = model.generate(prompt)
score = compute_task_metric(pred, gold) # metric unspecified (acc/f1)

# Hallucination Index (FinTerms-MCQ)
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
HI = correct / len(gold)

# Human Evaluation (FinTerms-Gen)
labels = human_annotate(predictions) # A, B, C, D
# Reported as distribution across categories

# Finance Bench
correct = sum(1 for p, g in zip(predictions, gold) if is_correct(p, g))
accuracy = correct / len(gold)
```

## Common pitfalls

- The exact evaluation metric (accuracy vs. F1) for the seven text tasks is not explicitly stated in the text, only decimal scores are reported.
- FinTerms-Gen human evaluation only uses 128 out of 150 Q&A pairs where both annotators agreed, which may introduce selection bias.
- Finance Bench evaluation uses only the 150-question open-source sample, not the full 10,231-question proprietary dataset.

## Evidence (verbatim from paper)

> We derive a hallucinations index (HI), defined as the proportion of correctly generated definitions by each model (higher is better), based on the models’ performance in this MCQ task. As seen in Table [8], the models’ performances on the HI vary significantly.

## Citation

```bibtex
@misc{bhatia2024fintral,
  title={FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models},
  author={Bhatia et al. (2024)},
  year={2024},
  note={arXiv:2402.10986}
}
```

- arXiv: 2402.10986