Legal Text Classification Eval

---
name: legal-text-classification-eval
description: This benchmark assesses models on hierarchical legal text classification tasks (LAP, JP, CP) across three Swiss languages. It tests domain-specific classification accuracy and robustness to long legal documents and multilingual inputs. Use when the user wants to benchmark on Legal Classification (LAP, JP, CP), or asks about evaluating this task. Reports Hierarchical Macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.09237
  bibtex_key: stern2023legal
  confidence: high
---

# legal-text-classification-eval

> One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support — Stern et al. (2023) (arXiv:2306.09237, 2023)

## What this evaluates

This benchmark assesses models on hierarchical legal text classification tasks (LAP, JP, CP) across three Swiss languages. It tests domain-specific classification accuracy and robustness to long legal documents and multilingual inputs.

## Datasets

- **Legal Classification (LAP, JP, CP)** — total ?; splits: train (-1), val (1000), test (-1)

## Metrics

- `Hierarchical Macro-F1` **(primary)** — range: [0, 1]
  - Harmonic mean of macro-averaged F1 scores aggregated across languages, configurations, and datasets to penalize outlier low scores and promote fairness.

## Input / output format

**Input**: Legal documents (up to 4096 tokens) with hierarchical class labels, provided in German, French, or Italian.

**Output**: Predicted hierarchical class label(s) for each document.

## Scoring recipe

```python
def score_classification(predictions, golds, languages, configs, datasets):
    f1_scores = []
    for lang in languages:
        for cfg in configs:
            for ds in datasets:
                mask = (lang & cfg & ds)
                f1_scores.append(macro_f1(predictions[mask], golds[mask]))
    return harmonic_mean(f1_scores)
```

## Common pitfalls

- Zero-shot evaluation uses only one instruction and example, which may underperform compared to few-shot setups.
- Truncation at 4096 tokens can discard relevant legal context, artificially lowering classification F1.

## Evidence (verbatim from paper)

> For our TC tasks, namely LAP, JP, and CP, we adopted the LEXTREME benchmark setup [10], namely hierarchical aggregation of macro-averaged F1 scores using harmonic mean for fairness (the harmonic mean is biased more towards lower scores than the geometric or arithmetic mean).

## Citation

```bibtex
@misc{stern2023legal,
  title={One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support},
  author={Stern et al. (2023)},
  year={2023},
  note={arXiv:2306.09237}
}
```

- arXiv: 2306.09237