Docfinqa Eval

---
name: docfinqa-eval
description: Evaluates long-context financial reasoning and context retrieval capabilities. It probes whether models can accurately retrieve relevant sections from lengthy SEC reports and answer numerical questions grounded in those documents. Use when the user wants to benchmark on DocFinQA, or asks about evaluating this task. Reports HR@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.06915
  bibtex_key: reddy2024docfinqa
  confidence: high
---

# docfinqa-eval

> DocFinQA: A Long-Context Financial Reasoning Dataset — Varshini Reddy et al. (2024) (arXiv:2401.06915, 2024)

## What this evaluates

Evaluates long-context financial reasoning and context retrieval capabilities. It probes whether models can accurately retrieve relevant sections from lengthy SEC reports and answer numerical questions grounded in those documents.

## Datasets

- **DocFinQA** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `HR@k` **(primary)** — range: [0, 1]
  - Hit Rate at k: the fraction of queries where the ground-truth relevant chunk is ranked within the top-k retrieved chunks.
- `accuracy` — range: percent
  - Exact match accuracy of the generated answer against the ground-truth numerical answer or Python program.

## Input / output format

**Input**: For retrieval: a question q and a set of document chunks C. For QA: a question q and k retrieved chunks provided as in-context examples.

**Output**: For retrieval: the index or rank of the retrieved chunk. For QA: a natural language or numerical answer.

## Scoring recipe

```python
# Retrieval HR@k
hr_at_k = sum(1 for q in queries if target_chunk_idx in top_k_indices[q]) / len(queries)
# QA Accuracy
accuracy = sum(1 for q in queries if normalize(generated_answer[q]) == normalize(gold_answer[q])) / len(queries)
```

## Common pitfalls

- Retrieval performance is highly sensitive to the embedding model; fine-tuned ColBERT significantly outperforms pre-trained baselines.
- LLM accuracy strongly depends on the few-shot configuration and context length, with generic LLMs preferring shorter contexts + more shots, while code-trained LLMs prefer longer contexts.
- Proprietary models like GPT-3 and BloombergGPT were excluded due to API inaccessibility, limiting direct comparison with the latest commercial baselines.

## Evidence (verbatim from paper)

> We evaluate these models using HR@k on the test set of DocFinQA using the target $c^{igstar}.

## Citation

```bibtex
@misc{reddy2024docfinqa,
  title={DocFinQA: A Long-Context Financial Reasoning Dataset},
  author={Varshini Reddy et al. (2024)},
  year={2024},
  note={arXiv:2401.06915}
}
```

- arXiv: 2401.06915