Court Judgment Prediction Eval

---
name: court-judgment-prediction-eval
description: Tests a model's ability to predict binary case outcomes (accepted/denied) and generate human-readable explanations by citing relevant sentences from the input document. This probes joint reasoning, outcome forecasting, and justification generation in legal contexts. Use when the user wants to benchmark on LegalEval CJPE Dataset, or asks about evaluating this task. Reports standard F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.09548
  bibtex_key: modi2023legaleval
  confidence: high
---

# court-judgment-prediction-eval

> SemEval 2023 Task 6: LegalEval - Understanding Legal Texts — Modi et al. (2023) (arXiv:2304.09548, 2023)

## What this evaluates

Tests a model's ability to predict binary case outcomes (accepted/denied) and generate human-readable explanations by citing relevant sentences from the input document. This probes joint reasoning, outcome forecasting, and justification generation in legal contexts.

## Datasets

- **LegalEval CJPE Dataset** — total ?; splits: test (-1)

## Metrics

- `standard F1 score` **(primary)** — range: [0, 1]
  - Standard F1 score for binary classification of case outcomes (accepted vs. denied).
- `ROUGE-2 score` — range: [0, 1]
  - Recall-Oriented Understudy for Gisting Evaluation (ROUGE-2) measures the overlap of bigrams between the generated explanation and the gold explanation.

## Input / output format

**Input**: A legal judgment document.

**Output**: A binary label (accepted/denied) and a set of explanatory sentences extracted from the document.

## Scoring recipe

```python
def compute_cjpe_metrics(pred_label, pred_explanation, gold_label, gold_explanation):
    tp = 1 if pred_label == gold_label == 'accepted' else 0
    fp = 1 if pred_label == 'accepted' and gold_label == 'denied' else 0
    fn = 1 if pred_label == 'denied' and gold_label == 'accepted' else 0
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    rouge2 = compute_rouge_n(pred_explanation, gold_explanation, n=2)
    return {'f1': f1, 'rouge2': rouge2}
```

## Common pitfalls

- Explanation quality is hard to evaluate automatically; ROUGE-2 only measures n-gram overlap, not factual correctness or legal validity.
- Outcome prediction can be biased by dataset construction if not carefully debiased, leading to inflated scores on majority classes.

## Evidence (verbatim from paper)

> For the CJPE task, the evaluation for judgment prediction (binary classification) is done using the standard F1 score metric, and for the explanation sub-task, we use ROUGE-2 score

## Citation

```bibtex
@misc{modi2023legaleval,
  title={SemEval 2023 Task 6: LegalEval - Understanding Legal Texts},
  author={Modi et al. (2023)},
  year={2023},
  note={arXiv:2304.09548}
}
```

- arXiv: 2304.09548