Langid Web Crawl Eval

---
name: langid-web-crawl-eval
description: Evaluates language identification models on noisy web crawl data to measure their ability to accurately filter in-language sentences for low-resource languages. It probes domain mismatch and class imbalance effects on real-world LangID deployment. Use when the user wants to benchmark on Web Crawl & Held-out Eval Set, or asks about evaluating this task. Reports precision.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.14571
  bibtex_key: caswell2020languageid
  confidence: high
---

# langid-web-crawl-eval

> Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus — Caswell et al. (2020) (arXiv:2010.14571, 2020)

## What this evaluates

Evaluates language identification models on noisy web crawl data to measure their ability to accurately filter in-language sentences for low-resource languages. It probes domain mismatch and class imbalance effects on real-world LangID deployment.

## Datasets

- **Web Crawl & Held-out Eval Set** — total ?; splits: test (-1)

## Metrics

- `precision` **(primary)** — range: percent
  - Percentage of in-language sentences in the filtered web crawl output, judged by human raters over a sample of 100 sentences per filtering method.
- `recall` — range: percent
  - Percentage of correctly identified in-language sentences on a held-out evaluation set with ground truth labels.

## Input / output format

**Input**: Raw text snippets (sentences) extracted from a noisy multilingual web crawl.

**Output**: Binary decision: keep (in-language) or discard (out-of-language).

## Scoring recipe

```python
# Precision (human-judged on crawl sample)
human_judgments = [1 if sentence is in-language else 0 for sentence in sample_100_sentences]
precision = sum(human_judgments) / len(human_judgments) * 100

# Recall (on held-out eval set)
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
recall = correct / len(gold_labels) * 100
```

## Common pitfalls

- Precision is not model-computed but human-judged on a small sample (100 sentences) from the crawl.
- Recall is measured on a separate held-out eval set, not the full crawl, because the crawl lacks ground truth labels.

## Evidence (verbatim from paper)

> For each example language, we report 1. the precision of the crawl (percent of in-language sentences), as judged by human raters over a sample of 100 sentences per filtering method, 2. the recall of this method on our held-out eval sets, and 3. the percentage of the crawl removed by this filtering method. (Keep in mind that, while the precision and % filtered rows are measured on the noisy web crawl, the recall is measured on the held-out eval set.)

## Citation

```bibtex
@misc{caswell2020languageid,
  title={Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus},
  author={Caswell et al. (2020)},
  year={2020},
  note={arXiv:2010.14571}
}
```

- arXiv: 2010.14571