Fintagging Eval

---
name: fintagging-eval
description: Evaluates large language models on structure-aware XBRL tagging for financial information. It probes two subtasks: numeric entity identification (FinNI) and fine-grained concept linking (FinCL) against the US-GAAP taxonomy, testing the model's ability to extract structured facts and align them with hierarchical financial concepts. Use when the user wants to benchmark on FinTagging, or asks about evaluating this task. Reports macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.20650
  bibtex_key: wang2025fintagging
  confidence: high
---

# fintagging-eval

> FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information — Wang et al. (2025) (arXiv:2505.20650, 2025)

## What this evaluates

Evaluates large language models on structure-aware XBRL tagging for financial information. It probes two subtasks: numeric entity identification (FinNI) and fine-grained concept linking (FinCL) against the US-GAAP taxonomy, testing the model's ability to extract structured facts and align them with hierarchical financial concepts.

## Datasets

- **FinTagging** — total ?; splits: test (-1); repo https://github.com/The-FinAI/FinTagging

## Metrics

- `macro-F1` **(primary)** — range: [0, 1]
  - Harmonic mean of macro-precision and macro-recall, computed as 2 * (P * R) / (P + R). Macro averaging treats all tags equally regardless of frequency.
- `micro-F1` — range: [0, 1]
  - Harmonic mean of micro-precision and micro-recall, computed as 2 * (P * R) / (P + R). Micro averaging weights each instance equally, reflecting performance on frequent labels.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly linked taxonomy concepts out of total instances.

## Input / output format

**Input**: Financial text and tables provided via a prompt template. For the FinCL subtask, a candidate list of taxonomy concepts is retrieved beforehand.

**Output**: Structured triplet (Tag, Fact, Type) for FinNI, and a single selected US-GAAP taxonomy concept from the candidate list for FinCL.

## Scoring recipe

```python
def compute_f1(precision, recall):
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Relying solely on precision for FinNI, as missing facts are more damaging than producing a few spurious ones.
- Treating XBRL tagging as single-step extreme classification, which causes all models to collapse to zero F1.
- Ignoring error propagation in the two-stage pipeline, where extraction mistakes cap downstream linking accuracy.

## Evidence (verbatim from paper)

> From a macro perspective, which emphasizes balanced performance across frequent and rare tags, large general-purpose LLMs clearly dominate. DeepSeek-V3, GPT-4o, and Llama-4-Scout achieve the strongest macro-F1 scores, surpassing all fine-tuned PLMs and indicating better generalization to long-tail concepts.

## Citation

```bibtex
@misc{wang2025fintagging,
  title={FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.20650}
}
```

- arXiv: 2505.20650