Legal Zero Days Eval

---
name: legal-zero-days-eval
description: This benchmark probes an AI system's ability to detect previously undiscovered legal vulnerabilities within governance frameworks. It tests whether models can identify systemic flaws that could cause immediate disruption without requiring traditional litigation, measuring their capacity for advanced legal reasoning and regulatory logic parsing. Use when the user wants to benchmark on Legal Zero-Days, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.10050
  bibtex_key: sadler2025legalzerodays
  confidence: medium
---

# legal-zero-days-eval

> Legal Zero-Days: A Novel Risk Vector for Advanced AI Systems — Greg Sadler et al. (2025) (arXiv:2508.10050, 2025)

## What this evaluates

This benchmark probes an AI system's ability to detect previously undiscovered legal vulnerabilities within governance frameworks. It tests whether models can identify systemic flaws that could cause immediate disruption without requiring traditional litigation, measuring their capacity for advanced legal reasoning and regulatory logic parsing.

## Datasets

- **Legal Zero-Days** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly identified legal vulnerabilities out of the total number of vulnerabilities or puzzles presented. Calculated as (correct identifications / total instances) × 100.

## Input / output format

**Input**: A description of a legal puzzle or governance framework scenario containing an embedded, previously undiscovered legal vulnerability.

**Output**: A textual response identifying and describing the specific legal vulnerability within the provided scenario.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold):
    correct = 0
    for pred, gold_vuln in zip(predictions, gold):
        if is_correct_identification(pred, gold_vuln):
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- High performance variance across runs, indicated by large confidence intervals (e.g., ±13.50%), suggesting instability in vulnerability detection.
- Difficulty distinguishing between genuinely correct identifications and responses that merely mischaracterize or partially address the introduced flaws.
- Automated judge validation relied on a small ground-truth set (25 examples), which may not fully generalize to the broader benchmark.

## Evidence (verbatim from paper)

> Table 1 presents the performance of all evaluated models on our Legal Zero-Days benchmark. Gemini-2.5-pro-preview-05-06 achieved the highest accuracy at 10.00% ± 13.50%, followed by o3-2025-04-16 at 6.67% ± 9.70%. The remaining models performed considerably lower, with accuracy scores ranging from 1.85% to 5.19%.

## Citation

```bibtex
@misc{sadler2025legalzerodays,
  title={Legal Zero-Days: A Novel Risk Vector for Advanced AI Systems},
  author={Greg Sadler et al. (2025)},
  year={2025},
  note={arXiv:2508.10050}
}
```

- arXiv: 2508.10050