Financial Esg Nlp Bench Eval

---
name: financial-esg-nlp-bench-eval
description: Evaluates large language models on a suite of 13 financial and ESG-related NLP tasks, including sentiment analysis, classification, named entity recognition, relation extraction, financial question answering, text summarization, and sustainability report generation. It probes the model's ability to perform domain-specific reasoning, information extraction, and structured text generation in the financial sector. Use when the user wants to benchmark on FiQASA, FOMC, MultiFin, MLESG, NER, FINER-ORD, FinRED, SC, FinQA, TATQA, ConvFinQA, EDTSUM, TCFD-Bench, or asks about evaluating this task. Reports F1, MicroF1, EmAcc, Rouge-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.10906
  bibtex_key: wu2024susgengpt
  confidence: high
---

# financial-esg-nlp-bench-eval

> SusGen-GPT: A Data-Centric LLM for Financial NLP and Sustainability Report Generation — Qilong Wu et al. (arXiv:2412.10906, 2024)

## What this evaluates

Evaluates large language models on a suite of 13 financial and ESG-related NLP tasks, including sentiment analysis, classification, named entity recognition, relation extraction, financial question answering, text summarization, and sustainability report generation. It probes the model's ability to perform domain-specific reasoning, information extraction, and structured text generation in the financial sector.

## Datasets

- **FiQASA** — total ?; splits: test (-1)
- **FOMC** — total ?; splits: test (-1)
- **MultiFin** — total ?; splits: test (-1)
- **MLESG** — total ?; splits: test (-1)
- **NER** — total ?; splits: test (-1)
- **FINER-ORD** — total ?; splits: test (-1)
- **FinRED** — total ?; splits: test (-1)
- **SC** — total ?; splits: test (-1)
- **FinQA** — total ?; splits: test (-1)
- **TATQA** — total ?; splits: test (-1)
- **ConvFinQA** — total ?; splits: test (-1)
- **EDTSUM** — total ?; splits: test (-1)
- **TCFD-Bench** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall. Computed macro-averaged for classification and relation extraction tasks.
- `MicroF1` **(primary)** — range: [0, 1]
  - F1 score calculated by aggregating true positives, false positives, and false negatives across all classes before computing precision and recall.
- `EntityF1` — range: [0, 1]
  - F1 score computed over extracted entity spans, requiring exact boundary and type matching.
- `EmAcc` **(primary)** — range: [0, 1]
  - Exact match accuracy; the model's predicted answer must exactly match the gold answer string.
- `Rouge-1` — range: [0, 1]
  - Unigram overlap between generated and reference text, measured as F1 score.
- `BertScore` — range: [0, 1]
  - Cosine similarity between contextual embeddings (BERT) of generated and reference tokens, averaged across tokens.
- `Rouge-L` **(primary)** — range: [0, 1]
  - Longest common subsequence overlap between generated and reference text, measured as F1 score.
- `METEOR` — range: [0, 1]
  - Weighted F-score incorporating exact match, synonymy, stemming, and paraphrase matching.
- `BLEU-1` — range: [0, 1]
  - Unigram precision with a brevity penalty to discourage overly short generations.

## Input / output format

**Input**: Alpaca prompt template combining instruction, context, and question. For retrieval-augmented generation, top-10 relevant text chunks are retrieved using all-mpnet-base-v2 embeddings.

**Output**: Model-generated text: class labels, numerical answers, entity/relation spans, summaries, or structured TCFD-format ESG reports.

## Scoring recipe

```python
def evaluate(predictions, golds, task):
    if task in ['SA', 'RE']:
        return f1_score(golds, predictions, average='macro')
    elif task in ['HC', 'MLESG']:
        return f1_score(golds, predictions, average='micro')
    elif task == 'NER':
        return entity_f1(golds, predictions)
    elif task in ['FinQA', 'TATQA', 'ConvFinQA']:
        return exact_match(golds, predictions)
    elif task == 'SUM':
        return rouge1(golds, predictions), bertscore(golds, predictions)
    elif task == 'SRG':
        return rouge_l(golds, predictions), bertscore(golds, predictions), meteor(golds, predictions), bleu1(golds, predictions)
```

## Common pitfalls

- Different tasks use fundamentally different metrics (e.g., F1 vs EmAcc vs Rouge), making cross-task averaging invalid.
- TCFD-Bench is a newly proposed benchmark without a public HuggingFace ID or established baseline code, requiring custom evaluation scripts.
- Evaluation is conducted in zero-shot or few-shot mode despite the model being fine-tuned on SusGen-30K, so results reflect adaptation capability rather than raw base performance.

## Evidence (verbatim from paper)

> We evaluate and compare SusGen-GPT on eight tasks in total against other baseline models, including seven financial NLP tasks using well-established benchmarks as well sustainability report generation (SRG) using our proposed TCFD-Bench. In SA, it achieves an F1 score of 0.72 on the FiQASA dataset... For FinQA and FinTQA, the model scores 0.57 on FinQA... SusGen-GPT outperformed ChatReport in Rouge-L (0.20 vs. 0.14), BERTScore (0.40 vs. 0.32), and METEOR (0.27 vs. 0.12)...

## Citation

```bibtex
@misc{wu2024susgengpt,
  title={SusGen-GPT: A Data-Centric LLM for Financial NLP and Sustainability Report Generation},
  author={Qilong Wu et al.},
  year={2024},
  note={arXiv:2412.10906}
}
```

- arXiv: 2412.10906