Schema To Json Eval

---
name: schema-to-json-eval
description: Evaluates the ability of language models to extract structured information from heterogeneous tables (text, LaTeX, HTML, CSV, XML) using only a human-authored JSON schema as supervision. It probes schema-driven information extraction, testing attribute prediction accuracy across diverse domains and input formats without domain-specific labeled data. Use when the user wants to benchmark on MlTables, ChemTables, DisCoMat, SWDE, or asks about evaluating this task. Reports Table-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.14336
  bibtex_key: bai2023schema
  confidence: high
---

# schema-to-json-eval

> Schema-Driven Information Extraction from Heterogeneous Tables — Bai et al. (2023) (arXiv:2305.14336, 2023)

## What this evaluates

Evaluates the ability of language models to extract structured information from heterogeneous tables (text, LaTeX, HTML, CSV, XML) using only a human-authored JSON schema as supervision. It probes schema-driven information extraction, testing attribute prediction accuracy across diverse domains and input formats without domain-specific labeled data.

## Datasets

- **MlTables** — total ?; splits: train (-1), test (-1)
- **ChemTables** — total ?; splits: train (-1), test (-1)
- **DisCoMat** — total ?; splits: train (-1), test (-1)
- **SWDE** — total ?; splits: train (-1), test (-1)

## Metrics

- `Table-F1` **(primary)** — range: [0, 1]
  - Harmonic mean of attribute-level precision and recall. Precision is the ratio of correctly predicted attributes to total predicted attributes. Reported as macro-averaged across tables.
- `Token-level F1` — range: [0, 1]
  - Token-level F1 score for attribute value matching. A prediction is correct if the score exceeds a dataset-specific threshold tuned on the dev set to maximize alignment with human judgments.
- `Exact Match (EM)` — range: [0, 1]
  - Exact match of predicted attribute values against gold values.
- `Tuple-F1` — range: [0, 1]
  - Exact match of a predicted 4-element tuple against the gold tuple.
- `Page-F1` — range: [0, 1]
  - Fraction of pages where all attributes are accurately predicted.

## Input / output format

**Input**: Heterogeneous table data (text, LaTeX, HTML, CSV, XML) paired with a human-authored JSON schema template. Optional supplementary text (headers, captions) may be included.

**Output**: A JSON object conforming to the provided schema, containing extracted attribute values for each table.

## Scoring recipe

```python
def compute_table_f1(preds, golds):
    correct = 0
    total_pred = 0
    total_gold = 0
    for p, g in zip(preds, golds):
        matched = [k for k in p if k in g and token_f1(p[k], g[k]) > THRESHOLD]
        correct += len(matched)
        total_pred += len(p)
        total_gold += len(g)
    prec = correct / total_pred if total_pred else 0
    rec = correct / total_gold if total_gold else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) else 0
```

## Common pitfalls

- Removing table captions surprisingly improves performance because low-quality or non-specific captions confuse the model.
- SWDE evaluation focuses on identifying the correct HTML node containing the attribute value rather than exact text span matching, requiring token-level F1 for node selection.
- The threshold for token-level F1 is dataset-specific and must be tuned on the development set to maximize alignment with human judgments.

## Evidence (verbatim from paper)

> We introduce Table-F1, a new metric gauging overall attribute prediction performance within a table, for our two proposed datasets. Table-F1 represents the harmonic mean of precision and recall, with precision being the ratio of correctly predicted attributes to total predicted attributes. At the attribute level, we consider two metrics: token-level F1 and exact match (EM). For DisCoMat and SWDE, we use the metrics specified in the original papers. In the case of DisCoMat, we report Tuple-F1, where a predicted 4-element tuple is considered correct only if it exactly matches with the gold tuple. For SWDE, we report Page-F1, which measures the number of pages where the attributes are accurately predicted.

## Citation

```bibtex
@misc{bai2023schema,
  title={Schema-Driven Information Extraction from Heterogeneous Tables},
  author={Bai et al. (2023)},
  year={2023},
  note={arXiv:2305.14336}
}
```

- arXiv: 2305.14336