Gdibench Eval

---
name: gdibench-eval
description: Evaluates document intelligence by decoupling visual and reasoning complexity into graded difficulty levels (V0–V2, R0–R2). It probes a model’s ability to extract, reason over, and generalize across diverse document types while mitigating catastrophic forgetting during fine-tuning. Use when the user wants to benchmark on GDI-Bench, or asks about evaluating this task. Reports Accuracy / normalized edit distance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.00063
  bibtex_key: li2025gdibench
  confidence: medium
---

# gdibench-eval

> GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling — Siqi Li et al. (2025) (arXiv:2505.00063, 2025)

## What this evaluates

Evaluates document intelligence by decoupling visual and reasoning complexity into graded difficulty levels (V0–V2, R0–R2). It probes a model’s ability to extract, reason over, and generalize across diverse document types while mitigating catastrophic forgetting during fine-tuning.

## Datasets

- **GDI-Bench** — total ?; splits: test (-1)

## Metrics

- `Accuracy / normalized edit distance` **(primary)** — range: [0, 1]
  - For GDI-Bench, accuracy is computed as the percentage of correctly answered questions or extracted fields across vision (V0-V2) and reasoning (R0-R2) complexity levels. For cross-domain tasks (T1-T4), normalized edit distance measures the similarity between predicted and gold text sequences.

## Input / output format

**Input**: Document images paired with task-specific prompts or questions, processed through a standardized preprocessing pipeline.

**Output**: Text responses containing extracted information, answers to questions, or formatted document fields.

## Scoring recipe

```python
def compute_metric(predictions, golds, metric_type='accuracy'):
    if metric_type == 'accuracy':
        correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
        return correct / len(golds)
    elif metric_type == 'normalized_edit_distance':
        distances = [edit_distance(p, g) / max(len(p), len(g)) for p, g in zip(predictions, golds)]
        return sum(distances) / len(distances)
```

## Common pitfalls

- Difficulty levels (V0-V2, R0-R2) are decoupled, so models may excel in vision but fail in reasoning or vice versa.
- OmniDocBench uses a lower-is-better metric (↓), unlike other benchmarks in the suite.
- Cross-domain/cross-task evaluations (T1-T4) use normalized edit distance, not standard accuracy.

## Evidence (verbatim from paper)

> We compare the performance of Full-Parameter Fine-Tuning, LoRA Fine-Tuning, and the LW-AFT method under both settings, as shown in Table [4], which displays the normalized edit distance for each task.

## Citation

```bibtex
@misc{li2025gdibench,
  title={GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling},
  author={Siqi Li et al. (2025)},
  year={2025},
  note={arXiv:2505.00063}
}
```

- arXiv: 2505.00063