Rhetorical Roles Eval

---
name: rhetorical-roles-eval
description: Probes a model's ability to segment long, unstructured legal documents into semantically coherent units and assign each sentence a specific rhetorical role label (e.g., Facts, Ratio, Arguments). This capability is fundamental for downstream legal AI applications like summarization and precedent search. Use when the user wants to benchmark on LegalEval RR Dataset, or asks about evaluating this task. Reports weighted F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.09548
  bibtex_key: modi2023legaleval
  confidence: high
---

# rhetorical-roles-eval

> SemEval 2023 Task 6: LegalEval - Understanding Legal Texts — Modi et al. (2023) (arXiv:2304.09548, 2023)

## What this evaluates

Probes a model's ability to segment long, unstructured legal documents into semantically coherent units and assign each sentence a specific rhetorical role label (e.g., Facts, Ratio, Arguments). This capability is fundamental for downstream legal AI applications like summarization and precedent search.

## Datasets

- **LegalEval RR Dataset** — total ?; splits: test (-1)

## Metrics

- `weighted F1 score` **(primary)** — range: [0, 1]
  - Weighted average of per-class F1 scores, where weights correspond to the support (number of true instances) for each class.

## Input / output format

**Input**: A legal judgment document represented as a sequence of sentences.

**Output**: A sequence of rhetorical role labels, one for each input sentence.

## Scoring recipe

```python
def compute_weighted_f1(predictions, gold, classes):
    scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        scores.append(f1 * sum(1 for g in gold if g == cls))
    return sum(scores) / len(gold)
```

## Common pitfalls

- Class imbalance is significant in legal documents, making unweighted F1 misleading.
- Sentence boundary detection errors in raw text can misalign predictions with gold labels.

## Evidence (verbatim from paper)

> The rhetorical roles task (a multiclass prediction problem) is evaluated using a weighted F1 score based on the test data.

## Citation

```bibtex
@misc{modi2023legaleval,
  title={SemEval 2023 Task 6: LegalEval - Understanding Legal Texts},
  author={Modi et al. (2023)},
  year={2023},
  note={arXiv:2304.09548}
}
```

- arXiv: 2304.09548