Pubtables V2 Eval

---
name: pubtables-v2-eval
description: Evaluates vision-language and specialized models on page-level and document-level table structure recognition, requiring them to extract hierarchical table structures from full pages or multi-page documents. It also probes cross-page table continuation prediction by testing whether models can identify when a table spans two contiguous pages. Use when the user wants to benchmark on PubTables-v2, or asks about evaluating this task. Reports GriTS_Top.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.10888
  bibtex_key: smock2025pubtablesv2
  confidence: high
---

# pubtables-v2-eval

> PubTables-v2: A new large-scale dataset for full-page and multi-page table extraction — Smock et al. (2025) (arXiv:2512.10888, 2025)

## What this evaluates

Evaluates vision-language and specialized models on page-level and document-level table structure recognition, requiring them to extract hierarchical table structures from full pages or multi-page documents. It also probes cross-page table continuation prediction by testing whether models can identify when a table spans two contiguous pages.

## Datasets

- **PubTables-v2** — total 548414; splits: train (-1), test (-1)

## Metrics

- `GriTS_Top` **(primary)** — range: [0, 1]
  - Pseudo F1-score for cells that enforces global consistency across all rows and columns, independent of output format. Evaluated on top-level table structure.
- `GriTS_Con` — range: [0, 1]
  - Same as GriTS_Top but evaluated on cell content/continuation consistency.
- `Acc_Top` — range: [0, 1]
  - Percentage of tables where predicted top-level structure exactly matches ground truth.
- `Acc_Con` — range: [0, 1]
  - Percentage of tables where predicted cell content exactly matches ground truth.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall for binary cross-page table continuation classification.

## Input / output format

**Input**: Single page image, concatenated pair of contiguous page images, or full document (multiple pages).

**Output**: Table structure representation (e.g., HTML/graph) for each detected table, or binary label (continues/does not continue) for cross-page task.

## Scoring recipe

```python
def evaluate_page_level(pred_tables, gt_tables):
    # Match predictions to ground truth using Hungarian algorithm to maximize total GriTS
    matched_pairs = hungarian_match(pred_tables, gt_tables, score_fn=gripts_score)
    gripts_top = sum(gripts_score(p, g) for p, g in matched_pairs) / len(gt_tables)
    exact_match = sum(1 for p, g in matched_pairs if p == g) / len(gt_tables)
    return gripts_top, exact_match

def evaluate_cross_page(pred_labels, gt_labels):
    tp = sum(1 for p, g in zip(pred_labels, gt_labels) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(pred_labels, gt_labels) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(pred_labels, gt_labels) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- GriTS evaluates global table structure consistency rather than isolated cell overlaps, making it sensitive to row/column alignment errors.
- Page-level evaluation automatically matches predicted and ground-truth tables via the Hungarian algorithm, so models do not need explicit table correspondence labels.
- Cross-page negative samples are strictly drawn from contiguous pages within the same document to prevent models from exploiting stylistic shortcuts instead of learning visual continuation cues.

## Evidence (verbatim from paper)

> To measure TSR performance, we use GriTS metrics [smock2023grits] and exact match accuracy. GriTS can be interpreted as a pseudo F1-score for cells. One advantage of GriTS is that it enforces global consistency on the result across all rows and columns and is not tied to a particular format for representing the table. Exact match accuracy (Acc) is the percentage of tables for which the predictions and ground truth match exactly.

## Citation

```bibtex
@misc{smock2025pubtablesv2,
  title={PubTables-v2: A new large-scale dataset for full-page and multi-page table extraction},
  author={Smock et al. (2025)},
  year={2025},
  note={arXiv:2512.10888}
}
```

- arXiv: 2512.10888