Scrapegraphai 100k Eval

---
name: scrapegraphai-100k-eval
description: Evaluates LLM-based web information extraction by measuring structural validity (JSON parseability, schema compliance), key extraction accuracy (precision, recall, F1), and value extraction quality (type-aware exact match, BLEU) on real-world HTML-to-JSON tasks. Use when the user wants to benchmark on ScrapeGraphAI-100k, or asks about evaluating this task. Reports Key F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.15189
  bibtex_key: brach2026scrapegraphai100k
  confidence: high
---

# scrapegraphai-100k-eval

> ScrapeGraphAI-100k: A Large-Scale Dataset for LLM-Based Web Information Extraction — Brach et al. (2026) (arXiv:2602.15189, 2026)

## What this evaluates

Evaluates LLM-based web information extraction by measuring structural validity (JSON parseability, schema compliance), key extraction accuracy (precision, recall, F1), and value extraction quality (type-aware exact match, BLEU) on real-world HTML-to-JSON tasks.

## Datasets

- **ScrapeGraphAI-100k** — total 93695; splits: test (2714); HF `scrapegraphai/scrapegraph-100k-finetuning`; repo https://github.com/ScrapeGraphAI/scrapegraph-100k-paper

## Metrics

- `is_valid_json` — range: [0, 1]
  - Boolean metric indicating whether the model output can be successfully parsed as valid JSON.
- `is_schema_compliant` — range: [0, 1]
  - Boolean metric checking if the parsed JSON output conforms to the target schema structure.
- `Key precision` — range: [0, 1]
  - Precision of flattened JSON keys using dot-notation paths with [*] wildcards for arrays.
- `Key recall` — range: [0, 1]
  - Recall of flattened JSON keys using dot-notation paths with [*] wildcards for arrays.
- `Key F1` **(primary)** — range: [0, 1]
  - Harmonic mean of key precision and recall, measuring structural accuracy of extracted keys.
- `Missing keys` — range: other
  - Average number of gold keys absent from the model output per sample.
- `Extra keys` — range: other
  - Average number of hallucinated keys present in the model output but absent from gold per sample.
- `Value score` — range: [0, 1]
  - Type-aware metric averaging exact match for booleans/numbers, set equality for arrays, and sentence-level BLEU for strings.
- `Overall BLEU` — range: [0, 1]
  - Sentence-level BLEU score computed on the serialized JSON string of the model output.

## Input / output format

**Input**: Raw HTML content (DOM structure) and a natural-language prompt specifying the extraction target, along with a JSON schema defining the desired output structure.

**Output**: A JSON object conforming to the target schema, containing extracted keys and values from the HTML.

## Scoring recipe

```python
def evaluate(pred_str, gold_str, schema):
    pred = json.loads(pred_str) if is_valid_json(pred_str) else None
    gold = json.loads(gold_str)
    valid = pred is not None
    schema_ok = check_schema(pred, schema) if valid else False
    p_keys = flatten_keys(pred)
    g_keys = flatten_keys(gold)
    tp = len(p_keys & g_keys)
    prec = tp / len(p_keys) if p_keys else 0
    rec = tp / len(g_keys) if g_keys else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) else 0
    missing = len(g_keys - p_keys)
    extra = len(p_keys - g_keys)
    val_scores = [type_aware_match(pred[k], gold[k]) for k in g_keys if k in p_keys]
    value_score = mean(val_scores) if val_scores else 0
    overall_bleu = sentence_bleu(tokenize(pred_str), tokenize(gold_str))
    return valid, schema_ok, prec, rec, f1, missing, extra, value_score, overall_bleu
```

## Common pitfalls

- Models may produce structurally valid JSON that perfectly matches the schema but contains hallucinated or imprecise values, creating a structural-semantic gap.
- Key matching uses dot-notation with [*] wildcards for arrays; failing to account for array wildcards will artificially deflate precision/recall.
- Evaluation excludes samples exceeding the 8192-token context limit, so results may not generalize to very long documents.

## Evidence (verbatim from paper)

> Structural validity metrics assess whether outputs are well-formed: is_valid_json checks JSON parseability, while is_schema_compliant verifies conformance to the target schema. Key extraction metrics (precision, recall, F1) measure structural accuracy by comparing flattened JSON keys using dot-notation paths with [*] wildcards for arrays. Value extraction is captured by value_score, a type-aware metric averaging exact match for booleans/numbers, set equality for arrays, and sentence-level BLEU*(Papineni et al., [2002])* for strings. Finally, overall_bleu on serialized JSON provides a holistic quality measure.

## Citation

```bibtex
@misc{brach2026scrapegraphai100k,
  title={ScrapeGraphAI-100k: A Large-Scale Dataset for LLM-Based Web Information Extraction},
  author={Brach et al. (2026)},
  year={2026},
  note={arXiv:2602.15189}
}
```

- arXiv: 2602.15189