Fingen Eval

---
name: fingen-eval
description: Evaluates forward-looking argument generation in finance across text-to-claim, chart-to-argument, and news-to-argument tasks. Probes a model's ability to generate plausible, structured future scenarios and claims based on financial inputs while maintaining factual consistency and handling financial terminology and numerals. Use when the user wants to benchmark on FinGen, or asks about evaluating this task. Reports ROUGE-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.20708
  bibtex_key: chen2024fingen
  confidence: high
---

# fingen-eval

> FinGen: A Dataset for Argument Generation in Finance — Chung-Chi Chen et al. (2024) (arXiv:2405.20708, 2024)

## What this evaluates

Evaluates forward-looking argument generation in finance across text-to-claim, chart-to-argument, and news-to-argument tasks. Probes a model's ability to generate plausible, structured future scenarios and claims based on financial inputs while maintaining factual consistency and handling financial terminology and numerals.

## Datasets

- **FinGen** — total ?; splits: train (-1), test (-1)

## Metrics

- `ROUGE-1` **(primary)** — range: [0, 1]
  - F1 score of unigram overlap between generated and reference text.
- `ROUGE-2` — range: [0, 1]
  - F1 score of bigram overlap between generated and reference text.
- `ROUGE-L` — range: [0, 1]
  - F1 score based on the longest common subsequence between generated and reference text.
- `BERT Score` — range: [0, 1]
  - Cosine similarity between contextual embeddings of generated and reference text, averaged over tokens.

## Input / output format

**Input**: Task-dependent: (1) Text excerpts (e.g., earnings calls), (2) Financial charts/images, (3) News headlines or full articles.

**Output**: Forward-looking financial claims or arguments, typically starting with phrases like 'We expect' or 'We believe', containing financial terms and numerals.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    rouge = rouge_score(references, predictions, avg=True)  # ROUGE-1, 2, L
    bert = bert_score(references, predictions, lang='en')   # BERT Score F1
    return rouge, bert
```

## Common pitfalls

- Models heavily copy financial terms from the input rather than generating novel ones.
- Models systematically under-generate numerals compared to ground truth and exhibit numeral hallucination.
- Generated claims often rely on generic boilerplate phrases like 'We expect' or 'We believe' without substantive reasoning.

## Evidence (verbatim from paper)

> We adopt ROUGE metrics and BERT Scores for evaluating all results.

## Citation

```bibtex
@misc{chen2024fingen,
  title={FinGen: A Dataset for Argument Generation in Finance},
  author={Chung-Chi Chen et al. (2024)},
  year={2024},
  note={arXiv:2405.20708}
}
```

- arXiv: 2405.20708