Finlmeval Eval

---
name: finlmeval-eval
description: Evaluates the financial natural language processing capabilities of encoder-only and decoder-only language models across multiple classification tasks. It probes zero-shot prompting, in-context learning strategies, and the impact of data availability (public vs. proprietary) on model performance. Use when the user wants to benchmark on FinSent, FPB, FiQA SA, ESG, FLS, QA, Headlines-PDU, Headlines-PDC, Headlines-PDD, Headlines-PI, Headlines-AC, Headlines-FI, Headlines-PS, NER, FOMC, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.12664
  bibtex_key: guo2023ischatgpt
  confidence: medium
---

# finlmeval-eval

> Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing — Guo et al. (2023) (arXiv:2310.12664, 2023)

## What this evaluates

Evaluates the financial natural language processing capabilities of encoder-only and decoder-only language models across multiple classification tasks. It probes zero-shot prompting, in-context learning strategies, and the impact of data availability (public vs. proprietary) on model performance.

## Datasets

- **FinSent** — total ?; splits: train (-1), val (-1), test (-1)
- **FPB** — total ?; splits: train (-1), val (-1), test (-1)
- **FiQA SA** — total ?; splits: train (-1), val (-1), test (-1)
- **ESG** — total ?; splits: train (-1), val (-1), test (-1)
- **FLS** — total ?; splits: train (-1), val (-1), test (-1)
- **QA** — total ?; splits: train (-1), val (-1), test (-1)
- **Headlines-PDU** — total ?; splits: train (-1), val (-1), test (-1)
- **Headlines-PDC** — total ?; splits: train (-1), val (-1), test (-1)
- **Headlines-PDD** — total ?; splits: train (-1), val (-1), test (-1)
- **Headlines-PI** — total ?; splits: train (-1), val (-1), test (-1)
- **Headlines-AC** — total ?; splits: train (-1), val (-1), test (-1)
- **Headlines-FI** — total ?; splits: train (-1), val (-1), test (-1)
- **Headlines-PS** — total ?; splits: train (-1), val (-1), test (-1)
- **NER** — total ?; splits: train (-1), val (-1), test (-1)
- **FOMC** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Not explicitly named in the text; values in [0,1] suggest Accuracy or F1-score. Computed as the proportion of correctly predicted labels over total instances per task.

## Input / output format

**Input**: Text snippets (sentences or headlines) for classification tasks. For in-context learning, prompts are constructed by appending selected demonstration examples (random or similarity-based) to the query.

**Output**: Predicted class label.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Data leakage risks on proprietary datasets (FinSent, ESG, FLS, QA) can artificially inflate zero-shot LLM performance if not strictly controlled.
- In-context learning effectiveness is highly prompt-dependent; random vs. similarity-based demonstration selection yields divergent results across tasks.
- Assuming decoder-only LLMs universally outperform fine-tuned encoder-only models ignores task-specific data scarcity and imbalance issues.

## Evidence (verbatim from paper)

> Table 2 compares the results of the fine-tuned encoder-only models and zero-shot decoder-only models in 9 financial datasets. We have the following findings: In 6 out of 9 datasets, fine-tuned encoder-only models can perform better than decoder-only models.

## Citation

```bibtex
@misc{guo2023ischatgpt,
  title={Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing},
  author={Guo et al. (2023)},
  year={2023},
  note={arXiv:2310.12664}
}
```

- arXiv: 2310.12664