Financial Phrase Bank Sentiment Eval

---
name: financial-phrase-bank-sentiment-eval
description: Probes the ability of LLMs and traditional NLP tools to accurately classify financial sentiment (positive, neutral, or negative) from news headlines and earnings-related text. It specifically evaluates how well models capture nuanced, hedged, or domain-specific financial language compared to baseline sentiment engines. Use when the user wants to benchmark on Financial Phrase Bank, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.16090
  bibtex_key: kubica2025financialnuance
  confidence: high
---

# financial-phrase-bank-sentiment-eval

> Can AI Read Between The Lines? Benchmarking LLMs On Financial Nuance — Kubica et al. (2025) (arXiv:2505.16090, 2025)

## What this evaluates

Probes the ability of LLMs and traditional NLP tools to accurately classify financial sentiment (positive, neutral, or negative) from news headlines and earnings-related text. It specifically evaluates how well models capture nuanced, hedged, or domain-specific financial language compared to baseline sentiment engines.

## Datasets

- **Financial Phrase Bank** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correct sentiment classifications (positive, neutral, or negative) against the pre-labeled gold standard across all evaluated sentences.

## Input / output format

**Input**: Financial sentences or headlines from the dataset, preprocessed for traditional NLP libraries, or fed via identical prompts to LLM-based tools.

**Output**: A single sentiment label: positive, neutral, or negative.

## Scoring recipe

```python
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() in ['positive', 'neutral', 'negative'] and pred.strip().lower() == gold.strip().lower():
        correct += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- LLM outputs often contain conversational filler or formatting; strict label extraction is required to avoid false negatives.
- Traditional NLP tools (e.g., TextBlob) default to neutral sentiment and miss domain-specific cues, requiring careful configuration or prompt engineering.
- Structured data (CSVs) fed to LLMs can cause hallucinations or formatting errors; converting to plain text is necessary for reliable evaluation.

## Evidence (verbatim from paper)

> After each model returned the sentiment of each sentence, accuracy was measured as the percentage of correct classifications against the pre-labeled dataset.

## Citation

```bibtex
@misc{kubica2025financialnuance,
  title={Can AI Read Between The Lines? Benchmarking LLMs On Financial Nuance},
  author={Kubica et al. (2025)},
  year={2025},
  note={arXiv:2505.16090}
}
```

- arXiv: 2505.16090