Sfiog Eval

---
name: sfiog-eval
description: Evaluates large language models' ability to generate coherent, logically structured financial investment opinions based on company context and questions. It probes reasoning over mere knowledge retrieval by testing performance across varying degrees of familiarity and novelty in companies and questions. Use when the user wants to benchmark on sFIOG, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.01505
  bibtex_key: son2023beyondclassification
  confidence: high
---

# sfiog-eval

> Beyond Classification: Financial Reasoning in State-of-the-Art Language Models — Son et al. (2023) (arXiv:2305.01505, 2023)

## What this evaluates

Evaluates large language models' ability to generate coherent, logically structured financial investment opinions based on company context and questions. It probes reasoning over mere knowledge retrieval by testing performance across varying degrees of familiarity and novelty in companies and questions.

## Datasets

- **sFIOG** — total 11802; splits: test (-1); repo https://github.com/guijinSON/FIOG

## Metrics

- `ROUGE-L` **(primary)** — range: [0, 1]
  - Recall-Oriented Understudy for Gisting Evaluation using the longest common subsequence. Measures the overlap of the longest matching word sequence between the generated text and the reference answer.
- `BERTScore` — range: [0, 1]
  - Computes semantic similarity by matching tokens in the generated and reference texts using contextual embeddings from a pre-trained BERT model, then calculating precision, recall, and F1 scores.
- `ROUGE-2` — range: [0, 1]
  - Measures the overlap of 2-grams (bigrams) between the generated text and the reference answer.

## Input / output format

**Input**: Company context (provided as full-text or Q&A format) paired with a specific question requiring an investment opinion.

**Output**: A generated investment opinion/thesis text, constrained to a maximum of 512 new tokens.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    rouge_l = rouge_score(references, predictions, rouge_types=['rougeL'])['rougeL.fmeasure']
    rouge_2 = rouge_score(references, predictions, rouge_types=['rouge2'])['rouge2.fmeasure']
    bert_f1 = bert_score.score(predictions, references, lang='en')[2]
    return {
        'ROUGE-L': rouge_l,
        'ROUGE-2': rouge_2,
        'BERTScore': bert_f1.mean()
    }
```

## Common pitfalls

- Models are constrained to exactly 512 max new tokens during decoding to prevent longer outputs from artificially inflating overlap metrics.
- LLM-based automated evaluators were explicitly excluded because they were found to misalign with human judgments.
- Type#2 questions (known companies, novel Q&A pairs) proved harder for models than Type#3 (novel companies), contrary to initial assumptions about knowledge generalization.

## Evidence (verbatim from paper)

> Automated metrics included rouge-2 and rougeL [Lin, 2004], measuring text overlap, and BERTScore [Zhang et al., 2019], assessing semantic similarity. ... LLama demonstrates superior performance, achieving the highest average scores in ROUGE-L (0.217) and BERTScore (0.821).

## Citation

```bibtex
@misc{son2023beyondclassification,
  title={Beyond Classification: Financial Reasoning in State-of-the-Art Language Models},
  author={Son et al. (2023)},
  year={2023},
  note={arXiv:2305.01505}
}
```

- arXiv: 2305.01505