Financial Nlp Eval

---
name: financial-nlp-eval
description: Evaluates the capability of large language models (ChatGPT and GPT-4) and domain-specific models to solve a variety of financial text analytics tasks. It probes performance across sentiment analysis, classification, information extraction, and question answering, measuring how well models handle domain-specific knowledge and structured prediction. Use when the user wants to benchmark on Financial NLP Tasks (Sentiment, Classification, NER, RE, QA), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.05862
  bibtex_key: li2023arechatgpt
  confidence: medium
---

# financial-nlp-eval

> Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks — Li et al. (2023) (arXiv:2305.05862, 2023)

## What this evaluates

Evaluates the capability of large language models (ChatGPT and GPT-4) and domain-specific models to solve a variety of financial text analytics tasks. It probes performance across sentiment analysis, classification, information extraction, and question answering, measuring how well models handle domain-specific knowledge and structured prediction.

## Datasets

- **Financial NLP Tasks (Sentiment, Classification, NER, RE, QA)** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correct predictions out of total instances.
- `macro-F1 score` — range: [0, 1]
  - Unweighted mean of per-class F1 scores, treating all classes equally regardless of support.
- `weighted F1 score` — range: [0, 1]
  - Mean of per-class F1 scores weighted by the number of true instances for each class.
- `entity-level F1 score` — range: [0, 1]
  - F1 score calculated at the entity span level for NER, requiring exact match of predicted and gold entity boundaries and types.

## Input / output format

**Input**: Raw financial text, news, social media posts, filings, or reports. For QA, questions with context. Prompts are formatted for zero-shot, few-shot, or Chain-of-Thought (CoT) inference.

**Output**: Model-generated predictions: class labels, sentiment scores, entity spans, relation triples, or textual answers.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    acc = sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    # Compute precision, recall, F1 per class
    # macro-F1 = mean of per-class F1s
    # weighted-F1 = sum(support_i * F1_i) / total_samples
    # For NER: count exact span matches for precision/recall
    return acc, macro_f1, weighted_f1
```

## Common pitfalls

- The paper evaluates multiple distinct task types but does not specify a single headline metric; accuracy, macro-F1, and weighted-F1 are all reported without clear hierarchy.
- Dataset names, exact sizes, and train/val/test splits are not provided in the main text and are deferred to Table 1 and Appendix B, making independent replication difficult.

## Evidence (verbatim from paper)

> We use accuracy, macro-F1 score, and weighted F1 score (Wu et al., 2023) as the evaluation metrics. For the NER task, we calculate the entity-level F1 score.

## Citation

```bibtex
@misc{li2023arechatgpt,
  title={Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2305.05862}
}
```

- arXiv: 2305.05862