Coliee Task4 Legal Qa Eval

---
name: coliee-task4-legal-qa-eval
description: Evaluates large language models' ability to perform legal textual entailment and question answering in monolingual and cross-lingual settings. It probes how well models handle linguistic and structural disparities between English and Japanese legal contexts and questions. Use when the user wants to benchmark on COLIEE Task 4, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.18098
  bibtex_key: nguyen2024gpts
  confidence: high
---

# coliee-task4-legal-qa-eval

> GPTs and Language Barrier: A Cross-Lingual Legal QA Examination — Nguyen et al. (2024) (arXiv:2403.18098, 2024)

## What this evaluates

Evaluates large language models' ability to perform legal textual entailment and question answering in monolingual and cross-lingual settings. It probes how well models handle linguistic and structural disparities between English and Japanese legal contexts and questions.

## Datasets

- **COLIEE Task 4** — total 429; splits: full (429)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted binary answers (Y or N) out of the total number of instances.

## Input / output format

**Input**: Legal context (articles) followed by a question, formatted with language-specific headers.

**Output**: Binary answer: 'Y' or 'N'.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() in ['Y', 'N'] and pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = correct / len(predictions)
```

## Common pitfalls

- Cross-lingual settings (EN-JA, JA-EN) introduce linguistic and structural mismatches that degrade performance compared to monolingual baselines.
- Models must strictly output 'Y' or 'N' without explanation; verbose outputs will fail exact-match scoring.
- Context and question lengths vary significantly across years, potentially affecting tokenization and model attention.

## Evidence (verbatim from paper)

> We formatted the input prompt as follows for monolingual prompting: Prompt in English: {context} Question: {question} Answer (Y or N), no explain. In our experiments, we explore different combinations of context and question languages, yielding four distinct settings: English context and English question (EN-EN), Japanese context and Japanese question (JA-JA), and two cross-lingual settings: English context with Japanese question (EN-JA) and Japanese context with English question (JA-EN).

## Citation

```bibtex
@misc{nguyen2024gpts,
  title={GPTs and Language Barrier: A Cross-Lingual Legal QA Examination},
  author={Nguyen et al. (2024)},
  year={2024},
  note={arXiv:2403.18098}
}
```

- arXiv: 2403.18098