Mudabench Eval

---
name: mudabench-eval
description: MuDABench probes multi-document analytical QA capabilities, requiring models to synthesize quantitative insights and perform inter-document reasoning across large collections of heterogeneous financial documents. It specifically tests cross-document filtering, conditional information extraction, and multi-step numerical computation beyond standard single-document retrieval. Use when the user wants to benchmark on MuDABench, or asks about evaluating this task. Reports final-answer accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.22239
  bibtex_key: li2026mudabench
  confidence: high
---

# mudabench-eval

> Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA — Li et al. (2026) (arXiv:2604.22239, 2026)

## What this evaluates

MuDABench probes multi-document analytical QA capabilities, requiring models to synthesize quantitative insights and perform inter-document reasoning across large collections of heterogeneous financial documents. It specifically tests cross-document filtering, conditional information extraction, and multi-step numerical computation beyond standard single-document retrieval.

## Datasets

- **MuDABench** — total ?; splits: test (-1); repo https://github.com/Zhanli-Li/MuDABench

## Metrics

- `final-answer accuracy` **(primary)** — range: [0, 1]
  - Accuracy_final = (1/|Q|) * sum(T_i), where T_i is 1 if the model's predicted answer is semantically equivalent to the gold answer (judged by an LLM), else 0.
- `process accuracy` — range: [0, 1]
  - Accuracy_process = (1/|Q|) * sum(P_i). For standard RAG, P_i uses conservative coverage min(C_i, 1-E_i) based on LLM-judged semantic matches to gold facts. For document-grounded workflows, P_i is cell-wise accuracy |C_hat_i|/|C_i| on aligned rows.
- `full accuracy` — range: [0, 1]
  - Accuracy_full = (1/|Q|) * sum(m_i * T_i), where m_i is 1 if process accuracy P_i equals 1, and T_i is 1 if the final answer is correct.

## Input / output format

**Input**: A natural language question $Q_j$ and a collection of $k$ financial documents (annual reports, announcements, ESG reports) with associated metadata (ticker, fiscal year, document type).

**Output**: A final answer $A_i$ (numerical or textual), and optionally intermediate extracted facts or table cells.

## Scoring recipe

```python
def score_final(gold_ans, pred_ans):
    return 1 if llm_judge_semantically_equivalent(pred_ans, gold_ans) else 0

def score_process_rag(gold_facts, pred_facts):
    C = len(intersect_semantically(pred_facts, gold_facts)) / len(gold_facts)
    E = count_missing_or_wrong(pred_facts, gold_facts) / len(gold_facts)
    return min(C, 1 - E)

def score_process_cell(gold_cells, pred_cells):
    return len(intersect_exact(pred_cells, gold_cells)) / len(gold_cells)

# Aggregate
final_acc = mean(score_final(g, p) for g, p in dataset)
process_acc = mean(score_process_rag(g, p) for g, p in dataset)  # or cell variant
full_acc = mean((process_score == 1.0) * score_final(g, p) for g, p in dataset)
```

## Common pitfalls

- Process coverage can be less reliable when equivalent evidence can be expressed in multiple non-atomic fact forms, leading to potential underestimation of intermediate extraction quality.
- The evaluation relies heavily on an LLM-as-a-judge for semantic equivalence, which may overestimate coverage or introduce inconsistency compared to exact string matching.
- Standard RAG and document-grounded workflows require different scoring formulas (coverage vs. cell-wise accuracy); applying the wrong recipe yields invalid process accuracy scores.

## Evidence (verbatim from paper)

> We evaluate each system with three metrics: process accuracy, final-answer accuracy, and full accuracy. Among them, final-answer accuracy is our primary end-task metric, while process accuracy is mainly used as a diagnostic signal for intermediate extraction quality. We note that process coverage can be less reliable when equivalent evidence can be expressed in multiple non-atomic fact forms. Final-answer accuracy. For each question $Q_{i}\in\mathcal{Q}$, let $A_{i}$ be the gold final answer and $\hat{A}_{i}$ be the model prediction. Let $T_{i}\in{0,1}$ denote whether $\hat{A}_{i}$ is semantically equivalent to $A_{i}$ (judged by an LLM): Accuracy_final = (1/|Q|) \sum_{i}T_{i.

## Citation

```bibtex
@misc{li2026mudabench,
  title={Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.22239}
}
```

- arXiv: 2604.22239