Swiss Judgment Prediction Eval

---
name: swiss-judgment-prediction-eval
description: Evaluates Legal Judgment Prediction (LJP) models across languages (German, French, Italian), regions, and legal domains. It probes cross-lingual, cross-regional, and cross-domain transfer capabilities, as well as the impact of data augmentation and adapter-based fine-tuning on model fairness and accuracy. Use when the user wants to benchmark on SJP (Swiss Judgment Prediction), or asks about evaluating this task. Reports macro-averaged F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.12325
  bibtex_key: niklaus2022empirical
  confidence: high
---

# swiss-judgment-prediction-eval

> An Empirical Study on Cross-X Transfer for Legal Judgment Prediction — Niklaus et al. (2022) (arXiv:2209.12325, 2022)

## What this evaluates

Evaluates Legal Judgment Prediction (LJP) models across languages (German, French, Italian), regions, and legal domains. It probes cross-lingual, cross-regional, and cross-domain transfer capabilities, as well as the impact of data augmentation and adapter-based fine-tuning on model fairness and accuracy.

## Datasets

- **SJP (Swiss Judgment Prediction)** — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/JoelNiklaus/Swiss

## Metrics

- `macro-averaged F1 score` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score computed across the approval and dismissal classes to account for high class-imbalance (~20/80 ratio).

## Input / output format

**Input**: Legal case documents (up to 2048 tokens), processed as consecutive 512-token blocks for Hierarchical BERT encoding.

**Output**: Binary classification label: approval or dismissal.

## Scoring recipe

```python
def compute_macro_f1(gold, preds):
    # Calculate precision and recall for each class
    # Average them across classes, then compute F1
    return f1_score(gold, preds, average='macro')
```

## Common pitfalls

- The dataset has high class imbalance (~20/80 approval/dismissal ratio), so accuracy is misleading; macro-F1 is strictly required.
- Documents exceed 512 tokens (90% are up to 2048), requiring hierarchical block encoding (4x512) instead of standard BERT truncation.
- Results are averaged across 3 random seeds; single-run scores are not comparable to reported values.

## Evidence (verbatim from paper)

> We follow Niklaus et al. (2021) and report macro-averaged F1 score to account for the high class-imbalance in the dataset (approx. 20/80 approval/dismissal ratio). We repeat each experiment with 3 different random seeds and report the average score and standard deviation across runs (seeds).

## Citation

```bibtex
@misc{niklaus2022empirical,
  title={An Empirical Study on Cross-X Transfer for Legal Judgment Prediction},
  author={Niklaus et al. (2022)},
  year={2022},
  note={arXiv:2209.12325}
}
```

- arXiv: 2209.12325