Lexam Eval

---
name: lexam-eval
description: Probes large language models' ability to perform structured, multi-step legal reasoning on real-world law exam questions. It evaluates both open-ended legal analysis and multiple-choice selection across diverse jurisdictions and legal domains. Use when the user wants to benchmark on LEXam, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.12864
  bibtex_key: fan2025lexam
  confidence: high
---

# lexam-eval

> LEXam: Benchmarking Legal Reasoning on 340 Law Exams — Fan et al. (2025) (arXiv:2505.12864, 2025)

## What this evaluates

Probes large language models' ability to perform structured, multi-step legal reasoning on real-world law exam questions. It evaluates both open-ended legal analysis and multiple-choice selection across diverse jurisdictions and legal domains.

## Datasets

- **LEXam** — total 4886; splits: test (4886); repo https://github.com/LEXam-Benchmark/LEXam

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly selected answer labels out of total multiple-choice questions. Label distribution is balanced via permutation to ensure unbiased performance.
- `LLM-as-a-Judge score` — range: other
  - Minimum score assigned by an ensemble of GPT-4o, Qwen3-32B, and DeepSeek-V3 acting as judges. The judging prompt is expert-verified and calibrated to penalize plausible but incorrect legal information.

## Input / output format

**Input**: Open questions: course name/knowledge domain, standard legal reasoning guidance, and the exam question. MCQs: course title, standard legal reasoning steps guidance, and the question with four permuted answer choices.

**Output**: Open questions: free-text structured legal reasoning response. MCQs: single selected answer label.

## Scoring recipe

```python
# MCQ Evaluation
mcq_accuracy = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)

# Open Question Evaluation
def evaluate_open(model_output, question, domain):
    judge_prompt = build_judge_prompt(question, model_output, domain)
    scores = [llm_judge(judge_prompt) for llm_judge in [GPT4o, Qwen3_32B, DeepSeek_V3]]
    return min(scores)
```

## Common pitfalls

- Using lexical/semantic metrics (BLEU, ROUGE, BERTScore) for open questions, which fail to capture correct legal reasoning due to inherently low lexical similarity.
- Relying on a single LLM judge instead of the required minimum-score ensemble, which undermines reliability and fails the Alternative Annotator Test validation.
- Ignoring answer choice permutation in MCQs, which is necessary to balance label distribution and ensure accuracy reflects true model capability rather than positional bias.

## Evidence (verbatim from paper)

> We use accuracy scores for the MCQ evaluations. Since the choice label distribution is balanced through permutation, accuracy reflects unbiased performance of LLMs. ... In our evaluation, we adopt an minimum-score ensemble of GPT-4o, Qwen3-32B, and DeepSeek-V3 to grade open questions.

## Citation

```bibtex
@misc{fan2025lexam,
  title={LEXam: Benchmarking Legal Reasoning on 340 Law Exams},
  author={Fan et al. (2025)},
  year={2025},
  note={arXiv:2505.12864}
}
```

- arXiv: 2505.12864