Legal Bert Eval

---
name: legal-bert-eval
description: Evaluates domain-adapted BERT models on legal text classification and named entity recognition to measure the impact of further pre-training and hyperparameter tuning strategies. Use when the user wants to benchmark on EURLEX57K, ECHR-CASES, CONTRACTS-NER, or asks about evaluating this task. Reports accuracy, F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.02559
  bibtex_key: chalkidis2020legalbert
  confidence: medium
---

# legal-bert-eval

> LEGAL-BERT: The Muppets straight out of Law School — Chalkidis et al. (2020) (arXiv:2010.02559, 2020)

## What this evaluates

Evaluates domain-adapted BERT models on legal text classification and named entity recognition to measure the impact of further pre-training and hyperparameter tuning strategies.

## Datasets

- **EURLEX57K** — total ?; splits: train (-1), val (-1), test (-1)
- **ECHR-CASES** — total ?; splits: train (-1), val (-1), test (-1)
- **CONTRACTS-NER** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `accuracy, F1` **(primary)** — range: [0, 1]
  - Standard accuracy and F1 scores for multi-label classification and sequence tagging, as replicated from prior work (Chalkidis et al., 2019).

## Input / output format

**Input**: Raw legal text documents for classification tasks; tokenized text sequences for named entity recognition.

**Output**: Predicted class labels (binary or multi-label) for classification; predicted entity tags for sequence tagging.

## Scoring recipe

```python
# For classification tasks
acc = sum(pred == gold for pred, gold in zip(predictions, gold_labels)) / len(gold_labels)
# For NER tasks
f1 = compute_f1(predictions, gold_labels) # Standard BIO tag matching
# Aggregate across datasets/tasks as needed
```

## Common pitfalls

- Blindly following default BERT hyperparameters (learning rate, epochs, batch size) without domain-specific tuning.
- Using a fixed maximum number of training epochs (e.g., 4) which can cause underfitting on legal datasets; early stopping on validation loss is recommended.

## Evidence (verbatim from paper)

> We evaluate our models on text classification and sequence tagging using three datasets. EURLEX57K (Chalkidis et al., 2019b) is a large-scale multi-label text classification dataset of EU laws, also suitable for few and zero-shot learning. ECHR-CASES (Chalkidis et al., 2019a) contains cases from the European Court of Human Rights (Aletras et al., 2016) and can be used for binary and multi-label text classification. Finally, CONTRACTS-NER (Chalkidis et al., 2017, 2019d) is a dataset for named entity recognition on US contracts consisting of three subsets, contract header, dispute resolution, and lease details. We replicate the experiments of Chalkidis et al. (2019c,a,d) when fine-tuning BERT for all datasets.

## Citation

```bibtex
@misc{chalkidis2020legalbert,
  title={LEGAL-BERT: The Muppets straight out of Law School},
  author={Chalkidis et al. (2020)},
  year={2020},
  note={arXiv:2010.02559}
}
```

- arXiv: 2010.02559