Pdf Extraction Eval

---
name: pdf-extraction-eval
description: Evaluates open-source PDF information extraction tools across multiple content elements (metadata, references, tables, paragraphs, sections, etc.) on academic documents. It probes how well different tools handle layout-based segmentation, text extraction, and structural recognition in real-world academic PDFs. Use when the user wants to benchmark on DocBank, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.09957
  bibtex_key: meuschke2023pdfbenchmark
  confidence: high
---

# pdf-extraction-eval

> A Benchmark of PDF Information Extraction Tools using a Multi-Task and Multi-Domain Evaluation Framework for Academic Documents — Norman Meuschke et al. (arXiv:2303.09957, 2023)

## What this evaluates

Evaluates open-source PDF information extraction tools across multiple content elements (metadata, references, tables, paragraphs, sections, etc.) on academic documents. It probes how well different tools handle layout-based segmentation, text extraction, and structural recognition in real-world academic PDFs.

## Datasets

- **DocBank** — total ?; splits: test (-1); repo https://github.com/doc-analysis/DocBank

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). For metadata and general extraction tasks, a cumulative F1 score is reported, calculated as the sum of individual F1 scores across sub-elements (e.g., title, abstract, authors).

## Input / output format

**Input**: PDF files of academic documents (primarily from arXiv).

**Output**: Structured extraction of specific content elements per document, including title, abstract, authors, references, tables, paragraphs, sections, captions, footers, and equations.

## Scoring recipe

```python
def compute_f1(preds, gold):
    tp = len(set(preds) & set(gold))
    fp = len(set(preds) - set(gold))
    fn = len(set(gold) - set(preds))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * (prec * rec) / (prec + rec) if (prec + rec) > 0 else 0

def compute_cumulative_f1(element_f1s):
    return sum(element_f1s.values())
```

## Common pitfalls

- Cumulative F1 scores can mask poor performance on specific sub-elements (e.g., a tool might score well on title/abstract but poorly on authors).
- Table extraction tools often misidentify two-column layouts as tables or include captions in table regions, artificially lowering F1 scores.
- Tools may fail silently or throw PDF read exceptions, leading to discrepancies between '# Detected' and '# Processed' items in results tables.

## Evidence (verbatim from paper)

> Figure 6 shows the cumulative  $F_{1}$  scores of CERMINE, GROBID, PdFig, and Science Parse for the metadata extraction task, i.e., extracting title, abstract, and authors. Consequently, the best possible cumulative  $F_{1}$  score equals three.

## Citation

```bibtex
@misc{meuschke2023pdfbenchmark,
  title={A Benchmark of PDF Information Extraction Tools using a Multi-Task and Multi-Domain Evaluation Framework for Academic Documents},
  author={Norman Meuschke et al.},
  year={2023},
  note={arXiv:2303.09957}
}
```

- arXiv: 2303.09957