German Legal Qa Eval

---
name: german-legal-qa-eval
description: Evaluates large language models' ability to answer German legal questions accurately in both open-ended and multiple-choice formats. It probes factual grounding in noisy, real-world legal documents (LegalMC4) versus clean statutory text (BGB), and tests robustness to distractor information typical of retrieval-augmented generation (RAG) pipelines. Use when the user wants to benchmark on LegalMC4 QA, BGB QA, LegalMC4 MCQ, BGB MCQ, ARC (Easy/Challenge), ARC-DE, MMLU, or asks about evaluating this task. Reports LLM-judged factual correctness (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.14160
  bibtex_key: [REDACTED]
  confidence: high
---

# german-legal-qa-eval

> Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Law — Bashir et al. (2026) (arXiv:2601.14160, 2026)

## What this evaluates

Evaluates large language models' ability to answer German legal questions accurately in both open-ended and multiple-choice formats. It probes factual grounding in noisy, real-world legal documents (LegalMC4) versus clean statutory text (BGB), and tests robustness to distractor information typical of retrieval-augmented generation (RAG) pipelines.

## Datasets

- **LegalMC4 QA** — total 732; splits: test (732)
- **BGB QA** — total 715; splits: test (715)
- **LegalMC4 MCQ** — total 1315; splits: test (1315)
- **BGB MCQ** — total 1845; splits: test (1845)
- **ARC (Easy/Challenge)** — total ?; splits: test (-1)
- **ARC-DE** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)

## Metrics

- `LLM-judged factual correctness (%)` **(primary)** — range: percent
  - Percentage of open-ended questions where an LLM judge (GPT-4.1) rates the model's answer as factually correct relative to the gold reference, following standard LLM-as-a-judge protocols.
- `exact accuracy (%)` — range: percent
  - Percentage of multiple-choice questions where the model's predicted option exactly matches the gold label.

## Input / output format

**Input**: Open-ended questions grounded in provided legal passages (BGB sections or LegalMC4 documents with noisy context), or multiple-choice questions with a prompt and candidate options.

**Output**: For open-ended: a free-text answer grounded in the provided context. For multiple-choice: the selected option letter/text.

## Scoring recipe

```python
def score(predictions, golds, task_type):
    scores = []
    for pred, gold in zip(predictions, golds):
        if task_type == 'open_qa':
            # LLM judge (GPT-4.1) evaluates factual correctness
            is_correct = llm_judge(prompt=gold['question'], context=gold['context'], answer=pred, reference=gold['answer'])
            scores.append(1 if is_correct else 0)
        elif task_type == 'mcq':
            scores.append(1 if pred == gold['option'] else 0)
    return sum(scores) / len(scores) * 100
```

## Common pitfalls

- LegalMC4 test sets intentionally contain noisy, context-irrelevant passages mimicking RAG retrieval outputs; models must extract relevant facts rather than relying on clean statute text.
- Open-ended QA scoring relies on an LLM judge (GPT-4.1) rather than exact string matching, which can introduce judge bias or inconsistency if not carefully prompted.
- Performance on general benchmarks (ARC, MMLU) is reported as a secondary check for catastrophic forgetting, not as the primary evaluation target.

## Evidence (verbatim from paper)

> Two sets contain open-ended questions and are scored for factual correctness using an LLM judge (GPT-4.1), following standard LLM-based evaluation protocols (Zheng et al., [2023]; Liu et al., [2023]) which yielded in high correlations with human experts in similar scenarios (Hillebrand et al., [2024]). Two sets contain multiple-choice questions and are evaluated by exact accuracy.

## Citation

```bibtex
@misc{bashir2026domainadaptation,
  title={Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Law},
  author={Bashir et al. (2026)},
  year={2026},
  note={arXiv:2601.14160}
}
```

- arXiv: 2601.14160