Coliee Task4 Eval

---
name: coliee-task4-eval
description: Evaluates the ability of large language models to perform legal textual entailment, specifically measuring how model accuracy changes over time based on the year of the Japanese statute law data used. Use when the user wants to benchmark on COLIEE Task 4, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.05501
  bibtex_key: [REDACTED]
  confidence: medium
---

# coliee-task4-eval

> Black-Box Analysis: GPTs Across Time in Legal Textual Entailment Task — Nguyen et al. (2023) (arXiv:2309.05501, 2023)

## What this evaluates

Evaluates the ability of large language models to perform legal textual entailment, specifically measuring how model accuracy changes over time based on the year of the Japanese statute law data used.

## Datasets

- **COLIEE Task 4** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly predicted entailment labels out of the total number of instances. Calculated as correct predictions divided by total predictions.

## Input / output format

**Input**: Pairs of legal texts (premise and hypothesis) in either English or Japanese.

**Output**: Classification label indicating the entailment relationship (e.g., entailment, contradiction, neutral).

## Scoring recipe

```python
def calculate_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Performance fluctuates significantly across years due to shifts in legal language and dataset distribution, so single-year results may not generalize.
- Models show divergent performance between English and Japanese versions, heavily influenced by their respective training data distributions.
- Historical Japanese calendar years (Heisei/Reiwa) must be correctly mapped to Gregorian years for temporal analysis.

## Evidence (verbatim from paper)

> Table I presents a summary of the experimental results for GPT-3.5 and GPT-4 in both English and Japanese across the years. The table highlights each model's accuracy score for the English and Japanese data in the corresponding years.

## Citation

```bibtex
@misc{nguyen2023blackbox,
  title={Black-Box Analysis: GPTs Across Time in Legal Textual Entailment Task},
  author={Nguyen et al. (2023)},
  year={2023},
  note={arXiv:2309.05501}
}
```

- arXiv: 2309.05501