Dlue Eval

---
name: dlue-eval
description: Evaluates document language understanding across four core capabilities: classification, structural analysis, information extraction, and transcription. It probes models' ability to handle long documents, complex hierarchical structures, and dispersed knowledge spread across large contexts. Use when the user wants to benchmark on Hyperpartisan, ContractNLI, ECOM, RR, GUM, LitBank, NarrativeQA, SummScreen, GovReport, Qasper, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.09520
  bibtex_key: xu2023dlue
  confidence: high
---

# dlue-eval

> DLUE: Benchmarking Document Language Understanding — Xu et al. (2023) (arXiv:2305.09520, 2023)

## What this evaluates

Evaluates document language understanding across four core capabilities: classification, structural analysis, information extraction, and transcription. It probes models' ability to handle long documents, complex hierarchical structures, and dispersed knowledge spread across large contexts.

## Datasets

- **Hyperpartisan** — total ?; splits: test (-1)
- **ContractNLI** — total ?; splits: test (-1)
- **ECOM** — total ?; splits: test (-1)
- **RR** — total ?; splits: test (-1)
- **GUM** — total ?; splits: test (-1)
- **LitBank** — total ?; splits: test (-1)
- **NarrativeQA** — total ?; splits: test (-1)
- **SummScreen** — total ?; splits: test (-1)
- **GovReport** — total ?; splits: test (-1)
- **Qasper** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: percent
  - Standard F1 score computed as the harmonic mean of precision and recall. For classification tasks, accuracy is also reported. Values are averaged across three random seed repetitions.

## Input / output format

**Input**: Document text tokenized into sequences. For classification: [CLS] token prepended to the document. For structure analysis: sentence-level sequences with [CLS] tokens at the start of each sentence. For extraction: multi-span question-answering format. For transcription: encoder-decoder input format.

**Output**: Class label for classification; sequence labels for structure analysis; extracted text spans for information extraction; generated text for transcription.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == g)
    fp = sum(1 for p, g in zip(predictions, gold) if p != g and p in gold)
    fn = sum(1 for p, g in zip(predictions, gold) if g not in predictions)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- ContractNLI exhibits label bias related to document length, where longer contracts tend to entail hypotheses, artificially inflating performance on longer documents.
- No single architecture dominates all tasks; performance varies significantly across classification, structure, extraction, and transcription, requiring task-specific model selection.
- Long-range transformers may plateau in performance when document length exceeds input limits, failing to capture dispersed knowledge effectively.

## Evidence (verbatim from paper)

> human agreement on ECOM was measured at 80.8% F1 (Xu et al., 2022), much higher than our best baseline of 39.1% F1. Likewise, Dasigi et al. (2021) study a subset of Qasper that has multiple annotated answers, and find their overlap to be 60.9% F1, more than double our best baseline.

## Citation

```bibtex
@misc{xu2023dlue,
  title={DLUE: Benchmarking Document Language Understanding},
  author={Xu et al. (2023)},
  year={2023},
  note={arXiv:2305.09520}
}
```

- arXiv: 2305.09520