Publaynet Layout Eval

---
name: publaynet-layout-eval
description: Evaluates the ability of object detection models to identify and localize document layout elements (text, title, list, table, figure) in scientific PDF pages. It also probes transfer learning capabilities by fine-tuning on out-of-domain documents and table detection tasks. Use when the user wants to benchmark on PubLayNet, or asks about evaluating this task. Reports MAP @ IOU [0.50:0.95].
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1908.07836
  bibtex_key: zhong2019publaynet
  confidence: high
---

# publaynet-layout-eval

> PubLayNet: largest dataset ever for document layout analysis — Zhong et al. (2019) (arXiv:1908.07836, 2019)

## What this evaluates

Evaluates the ability of object detection models to identify and localize document layout elements (text, title, list, table, figure) in scientific PDF pages. It also probes transfer learning capabilities by fine-tuning on out-of-domain documents and table detection tasks.

## Datasets

- **PubLayNet** — total ?; splits: dev (-1), test (-1); repo https://github.com/ibm-aur-nlp/PubLayNet

## Metrics

- `MAP @ IOU [0.50:0.95]` **(primary)** — range: [0, 1]
  - Mean Average Precision averaged over Intersection over Union thresholds from 0.50 to 0.95 (step 0.05), following the COCO detection evaluation protocol.

## Input / output format

**Input**: PDF pages converted to images.

**Output**: Bounding boxes with class labels for five categories: Text, Title, List, Table, Figure.

## Scoring recipe

```python
def compute_map(predictions, ground_truth):
    aps = []
    for iou_th in np.arange(0.50, 0.96, 0.05):
        tp, fp = 0, 0
        for gt in ground_truth:
            best_pred = max(predictions, key=lambda p: iou(p, gt))
            if iou(best_pred, gt) >= iou_th and not best_pred.used:
                tp += 1; best_pred.used = True
            else:
                fp += 1
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / len(ground_truth)
        aps.append(interpolate_ap(prec, rec))
    return np.mean(aps)
```

## Common pitfalls

- Titles are consistently harder to detect than tables/figures due to smaller size and less distinctive shapes.
- Zero-shot transfer to out-of-domain documents (e.g., SPD) performs poorly; fine-tuning is required for comparable performance.
- Evaluation uses development and test sets, but exact split sizes are not provided in the paper.

## Evidence (verbatim from paper)

> The evaluation metric is the mean average precision (MAP) @ intersection over union (IOU) [0.50:0.95] of bounding boxes, which is used in the COCO competition. Both models can generate accurate (MAP > 0.9) document layout, where M-RCNN shows a small advantage over F-RCNN.

## Citation

```bibtex
@misc{zhong2019publaynet,
  title={PubLayNet: largest dataset ever for document layout analysis},
  author={Zhong et al. (2019)},
  year={2019},
  note={arXiv:1908.07836}
}
```

- arXiv: 1908.07836