Financial Retrieval Eval

---
name: financial-retrieval-eval
description: Evaluates the ability of text embedding models to retrieve relevant financial document passages given complex, long-form queries. It probes domain-specific retrieval capabilities, including sensitivity to company names, tickers, financial metrics, and date-specific information. Use when the user wants to benchmark on Financial Document Retrieval Dataset, or asks about evaluating this task. Reports Recall@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.07142
  bibtex_key: anderson2024greenback
  confidence: high
---

# financial-retrieval-eval

> Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt — Anderson et al. (2024) (arXiv:2411.07142, 2024)

## What this evaluates

Evaluates the ability of text embedding models to retrieve relevant financial document passages given complex, long-form queries. It probes domain-specific retrieval capabilities, including sensitivity to company names, tickers, financial metrics, and date-specific information.

## Datasets

- **Financial Document Retrieval Dataset** — total 15200000; splits: train (14300000), val (444000), test (447000)

## Metrics

- `Recall@1` **(primary)** — range: [0, 1]
  - Recall@1 measures the fraction of queries for which the ground truth passage is ranked first among the retrieved candidates. Computed as (number of queries with GT passage in top-1) / (total number of queries).

## Input / output format

**Input**: A natural language query (often long-form and complex) paired with a corpus of candidate text passages (max 512 tokens) augmented with a context line containing company name, ticker, and event/date.

**Output**: A ranked list of retrieved passages, with the primary metric computed on the top-1 result.

## Scoring recipe

```python
def compute_recall_at_1(predictions, gold):
    # predictions: list of top-1 passage IDs/text for each query
    # gold: list of ground truth passage IDs/text for each query
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Data contamination can occur if passages from the same source document are split across train/val/test; the protocol enforces document-level splitting to prevent this.
- Synthetic queries generated by LLMs may differ in distribution and complexity from real user queries, potentially inflating retrieval performance on held-out synthetic data.
- Passages lack crucial context (company name, ticker, date) unless explicitly prepended, which is required for both finetuning and evaluation.

## Evidence (verbatim from paper)

> Passage retrieval results: Recall@1 on a held-out test split of 447K query-passage pairs. BAM embeddings finetuned for financial document retrieval significantly outperform general-purpose embeddings.

## Citation

```bibtex
@misc{anderson2024greenback,
  title={Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt},
  author={Anderson et al. (2024)},
  year={2024},
  note={arXiv:2411.07142}
}
```

- arXiv: 2411.07142