Webmainbench Eval

---
name: webmainbench-eval
description: Evaluates a model's ability to extract main content from HTML web pages by classifying semantic blocks and generating clean text or Markdown. It probes robustness across varying difficulty levels and rich content types such as tables, code, and equations. Use when the user wants to benchmark on WebMainBench, WCEB, or asks about evaluating this task. Reports ROUGE-N F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.23119
  bibtex_key: liu2025dripper
  confidence: high
---

# webmainbench-eval

> Dripper: Token-Efficient Main HTML Extraction with a Lightweight LM — Liu et al. (2025) (arXiv:2511.23119, 2025)

## What this evaluates

Evaluates a model's ability to extract main content from HTML web pages by classifying semantic blocks and generating clean text or Markdown. It probes robustness across varying difficulty levels and rich content types such as tables, code, and equations.

## Datasets

- **WebMainBench** — total 7800; splits: test (7800)
- **WCEB** — total ?; splits: test (-1)

## Metrics

- `ROUGE-N F1` **(primary)** — range: [0, 1]
  - Standard ROUGE-N F1 score computed between the model's extracted text/markdown and the ground truth text. The paper reports the F1 measure for N-gram overlaps.

## Input / output format

**Input**: HTML document (raw or simplified), with a maximum context length of 32,000 tokens for the standard model.

**Output**: JSON classification of semantic blocks, which is subsequently converted to Markdown or plain text for evaluation. Inputs exceeding the context limit receive a score of 0.

## Scoring recipe

```python
def compute_metric(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        if pred is None or len(pred.strip()) == 0:
            scores.append(0.0)
        else:
            pred_text = convert_html_to_text(pred)
            gold_text = convert_html_to_text(gold)
            scores.append(rouge_f1(pred_text, gold_text))
    return sum(scores) / len(scores)
```

## Common pitfalls

- Inputs exceeding the 32k token context window are assigned a score of 0 for the standard model, which can artificially lower overall scores if not filtered or handled via fallback.
- Baseline tools output diverse formats (HTML+MD, MD, TEXT); evaluation requires consistent conversion to a common text format before computing ROUGE-N F1 to ensure fairness.

## Evidence (verbatim from paper)

> Table 2: Performance comparison on MainWebBench (ROUGE-N F1). Methods are categorized by Mode: Html+MD denotes tools outputting intermediate HTML converted to Markdown, while MD and TEXT denote native Markdown and Plain Text outputs, respectively.

## Citation

```bibtex
@misc{liu2025dripper,
  title={Dripper: Token-Efficient Main HTML Extraction with a Lightweight LM},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2511.23119}
}
```

- arXiv: 2511.23119