Greekbarbench Eval

---
name: greekbarbench-eval
description: This benchmark probes free-text legal reasoning and multi-hop statutory citation using Greek Bar exam questions. It requires models to analyze case facts, cite relevant Greek legal articles, and produce open-ended legal analysis. Performance is measured across three dimensions: factual accuracy, correct statutory citation, and quality of legal reasoning. Use when the user wants to benchmark on GreekBarBench, or asks about evaluating this task. Reports Mean.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.17267
  bibtex_key: [REDACTED]
  confidence: high
---

# greekbarbench-eval

> GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations — Chlapanis et al. (2025) (arXiv:2505.17267, 2025)

## What this evaluates

This benchmark probes free-text legal reasoning and multi-hop statutory citation using Greek Bar exam questions. It requires models to analyze case facts, cite relevant Greek legal articles, and produce open-ended legal analysis. Performance is measured across three dimensions: factual accuracy, correct statutory citation, and quality of legal reasoning.

## Datasets

- **GreekBarBench** — total 310; splits: test (310); repo https://github.com/nlpaueb/greek-bar-bench

## Metrics

- `Facts` — range: [1, 10]
  - 1–10 score based on a span-based rubric evaluating the accuracy of case facts mentioned in the response.
- `Articles` — range: [1, 10]
  - 1–10 score based on a span-based rubric evaluating the correct citation of Greek statutory articles.
- `Analysis` — range: [1, 10]
  - 1–10 score based on a span-based rubric evaluating the quality of legal reasoning and open-ended analysis.
- `Mean` **(primary)** — range: [1, 10]
  - Arithmetic average of the Facts, Articles, and Analysis scores.

## Input / output format

**Input**: System prompt instructing citation of Greek statutory articles, followed by a user prompt containing numbered case facts, a specific question, and relevant legal context (statutory articles and distractors).

**Output**: Free-text response containing legal analysis, case fact references, and citations to Greek statutory articles, formatted according to the benchmark's prompt template.

## Scoring recipe

```python
def evaluate(response, gold_facts, gold_articles, gold_analysis):
    judge_prompt = build_span_judge_prompt(response, gold_facts, gold_articles, gold_analysis)
    judge_output = call_llm_judge(judge_prompt, model='GPT-4.1-mini')
    facts_score = parse_score(judge_output, 'Facts')
    articles_score = parse_score(judge_output, 'Articles')
    analysis_score = parse_score(judge_output, 'Analysis')
    mean_score = (facts_score + articles_score + analysis_score) / 3
    return {'Facts': facts_score, 'Articles': articles_score, 'Analysis': analysis_score, 'Mean': mean_score}
```

## Common pitfalls

- LLM-judge scores are not directly comparable to human expert scores due to different dataset sizes (310 vs 22) and scoring methodologies.
- Model performance heavily depends on the legal context setting; evaluating in the 'no-context' setting tests parametric knowledge rather than reasoning with provided statutes, while 'oracle context' artificially inflates citation scores.
- The passing threshold is 6.00 across all dimensions, but models often fail specific dimensions (e.g., Articles/Analysis) even if the Mean passes.

## Evidence (verbatim from paper)

> We use GPT-4.1-mini as the judge with the Span-Judge prompt (§[3.2]), reporting the Facts score, Articles score, Analysis score and Mean score with standard error from three runs.

## Citation

```bibtex
@misc{chlapanis2025greekbarbench,
  title={GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations},
  author={Chlapanis et al. (2025)},
  year={2025},
  note={arXiv:2505.17267}
}
```

- arXiv: 2505.17267