Legalbenchpt Eval

---
name: legalbenchpt-eval
description: Evaluates large language models' ability to reason about and classify Portuguese legal concepts across 31 distinct legal domains. It probes zero-shot question-answering capabilities using multiple-choice, true/false, matching, and case-analysis formats derived from law exam questions. Use when the user wants to benchmark on LegalBench.PT, or asks about evaluating this task. Reports balanced accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.16357
  bibtex_key: canaverde2025legalbenchpt
  confidence: high
---

# legalbenchpt-eval

> LegalBench.PT: A Benchmark for Portuguese Law — Canaverde et al. (2025) (arXiv:2502.16357, 2025)

## What this evaluates

Evaluates large language models' ability to reason about and classify Portuguese legal concepts across 31 distinct legal domains. It probes zero-shot question-answering capabilities using multiple-choice, true/false, matching, and case-analysis formats derived from law exam questions.

## Datasets

- **LegalBench.PT** — total ?; splits: test (-1)

## Metrics

- `balanced accuracy` **(primary)** — range: [0, 1]
  - Average of recall scores across all classes (answer options). For binary true/false tasks, it equals standard accuracy.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall, used for multiple-selection and matching tasks where answers are sets of options.
- `Jaccard similarity` — range: [0, 1]
  - Size of intersection divided by size of union between predicted and gold option sets, used to compute agreement between human annotators.

## Input / output format

**Input**: A prompt specifying the legal area, detailed instructions on the question type (e.g., whether there is one or multiple correct answers), and the expected response format.

**Output**: Extracted letter options, true/false classifications, and matching pairs according to the instructed format.

## Scoring recipe

```python
def compute_metric(predictions, golds, q_type):
    if q_type in ['mc', 'cloze', 'case', 'tf']:
        recalls = []
        for opt in options:
            correct = sum(1 for p, g in zip(predictions, golds) if p == opt and g == opt)
            total_gold = sum(1 for g in golds if g == opt)
            recalls.append(correct / total_gold if total_gold > 0 else 0)
        return sum(recalls) / len(recalls)
    elif q_type in ['multi_select', 'matching']:
        preds_set = set(predictions)
        golds_set = set(golds)
        tp = len(preds_set & golds_set)
        fp = len(preds_set - golds_set)
        fn = len(golds_set - preds_set)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Ambiguity in legal questions: ~14.8% of questions may have multiple valid interpretations, making strict gold-standard matching unfair.
- Small question sets in niche fields (e.g., Maritime, Insolvency Law) yield less reliable performance estimates and do not represent comprehensive evaluations.
- Potential generation bias: The dataset was created using GPT-4o, which may artificially inflate its scores compared to other models.

## Evidence (verbatim from paper)

> We evaluate multiple-choice, cloze tasks, case analysis, and true/false questions separately using balanced accuracy. For multiple selection and matching questions, we use the $F_{1}$ score. We aggregate the results from the different quesion types and fields of law using a weighted average.

## Citation

```bibtex
@misc{canaverde2025legalbenchpt,
  title={LegalBench.PT: A Benchmark for Portuguese Law},
  author={Canaverde et al. (2025)},
  year={2025},
  note={arXiv:2502.16357}
}
```

- arXiv: 2502.16357