Legal Ner Eval

---
name: legal-ner-eval
description: Evaluates a model's capacity to identify and classify 14 domain-specific legal entities (e.g., Court, Statute, Precedent, Petitioner Name) within unstructured legal documents. This probes fine-grained information extraction capabilities tailored to legal terminology and structure. Use when the user wants to benchmark on LegalEval L-NER Dataset, or asks about evaluating this task. Reports standard F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.09548
  bibtex_key: modi2023legaleval
  confidence: high
---

# legal-ner-eval

> SemEval 2023 Task 6: LegalEval - Understanding Legal Texts — Modi et al. (2023) (arXiv:2304.09548, 2023)

## What this evaluates

Evaluates a model's capacity to identify and classify 14 domain-specific legal entities (e.g., Court, Statute, Precedent, Petitioner Name) within unstructured legal documents. This probes fine-grained information extraction capabilities tailored to legal terminology and structure.

## Datasets

- **LegalEval L-NER Dataset** — total ?; splits: test (-1)

## Metrics

- `standard F1 score` **(primary)** — range: [0, 1]
  - Standard token-level or span-level F1 score for named entity recognition, balancing precision and recall across all entity types.

## Input / output format

**Input**: A legal judgment document containing text with potential legal entities.

**Output**: A sequence of named entity tags (e.g., BIO format) or extracted spans with corresponding entity types.

## Scoring recipe

```python
def compute_ner_f1(predictions, gold):
    pred_spans = extract_spans(predictions)
    gold_spans = extract_spans(gold)
    tp = len(set(pred_spans) & set(gold_spans))
    fp = len(set(pred_spans) - set(gold_spans))
    fn = len(set(gold_spans) - set(pred_spans))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Legal entities often overlap or have nested structures, requiring careful span alignment.
- Standard NER models fail on domain-specific types like 'Statute' or 'Precedent' without legal fine-tuning.

## Evidence (verbatim from paper)

> For L-NER, we use standard F1 score metrics

## Citation

```bibtex
@misc{modi2023legaleval,
  title={SemEval 2023 Task 6: LegalEval - Understanding Legal Texts},
  author={Modi et al. (2023)},
  year={2023},
  note={arXiv:2304.09548}
}
```

- arXiv: 2304.09548