Financial Phrase Bank Eval

---
name: financial-phrase-bank-eval
description: This benchmark probes a model's ability to classify financial news sentences or phrases into positive, neutral, or negative semantic orientations. It specifically tests domain-specific sentiment analysis by evaluating how well models capture contextual cues, economic concepts, and directional event expectations in financial texts. Use when the user wants to benchmark on Financial PhraseBank, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1307.5336
  bibtex_key: maloe2013gooddebt
  confidence: high
---

# financial-phrase-bank-eval

> Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts — Malo et al. (2013) (arXiv:1307.5336, 2013)

## What this evaluates

This benchmark probes a model's ability to classify financial news sentences or phrases into positive, neutral, or negative semantic orientations. It specifically tests domain-specific sentiment analysis by evaluating how well models capture contextual cues, economic concepts, and directional event expectations in financial texts.

## Datasets

- **Financial PhraseBank** — total ?; splits: 100% agreement (2259), >75% agreement (3448), >66% agreement (4211), >50% agreement (4840)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified instances out of the total number of instances.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: Single sentences or phrases extracted from financial news articles.

**Output**: One of three class labels: Positive, Neutral, or Negative.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    accuracy = correct / len(gold)
    precisions, recalls = [], []
    for label in ['Positive', 'Neutral', 'Negative']:
        tp = sum(1 for p, g in zip(predictions, gold) if p == label and g == label)
        fp = sum(1 for p, g in zip(predictions, gold) if p == label and g != label)
        fn = sum(1 for p, g in zip(predictions, gold) if p != label and g == label)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        precisions.append(prec)
        recalls.append(rec)
    avg_prec = sum(precisions) / 3
    avg_rec = sum(recalls) / 3
    f1 = 2 * (avg_prec * avg_rec) / (avg_prec + avg_rec) if (avg_prec + avg_rec) > 0 else 0
    return accuracy, f1
```

## Common pitfalls

- The dataset is partitioned by inter-annotator agreement strength rather than a fixed train/validation/test split, requiring 10-fold cross-validation for evaluation.
- The class distribution is highly imbalanced, with Neutral sentences comprising roughly 60% of the data, which can artificially inflate accuracy if not monitored.
- Baseline models rely on different external lexicons (e.g., MPQA vs. Loughran & McDonald), making direct performance comparisons sensitive to lexicon quality rather than just algorithmic complexity.

## Evidence (verbatim from paper)

> Tables 4 and 5 show performance of the models on the four reference datasets defined based on the phrase bank with different degrees of inter-annotator agreement. The results reported for the algorithms with a machine learning component (i.e. MPQA and the two variants of LPS) are computed using 10-fold cross-validation. ... The accuracy levels achieved by the better performing LPS algorithm ranged from 0.828 to 0.951 on sentences with 100% agreement, and between 0.792 and 0.945 on the sentences with more than 75% agreement. Also the F1 score, which is defined as the harmonic mean of precision and recall was very high for LPS in comparison to the corresponding ranges for MPQA, W-Loughran and W-MPQA.

## Citation

```bibtex
@misc{maloe2013gooddebt,
  title={Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts},
  author={Malo et al. (2013)},
  year={2013},
  note={arXiv:1307.5336}
}
```

- arXiv: 1307.5336