Vlegal Bench Eval

---
name: vlegal-bench-eval
description: Evaluates large language models on Vietnamese legal reasoning within a civil law framework. It probes capabilities ranging from statutory recall and hierarchical navigation to multi-step conflict detection, penalty estimation, and ethical bias analysis. Use when the user wants to benchmark on VLegal-Bench, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.14554
  bibtex_key: nguyen2025vlegalbench
  confidence: high
---

# vlegal-bench-eval

> VLegal-Bench: Cognitively Grounded Benchmark for Vietnamese Legal Reasoning of Large Language Models — Nguyen Tien Dong et al. (arXiv:2512.14554, 2025)

## What this evaluates

Evaluates large language models on Vietnamese legal reasoning within a civil law framework. It probes capabilities ranging from statutory recall and hierarchical navigation to multi-step conflict detection, penalty estimation, and ethical bias analysis.

## Datasets

- **VLegal-Bench** — total 10450; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Proportion of correctly predicted answers out of total instances. Standard exact-match for multiple-choice and classification tasks.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall. Reported separately for positive (Y-F1) and negative (N-F1) classes in conflict detection tasks, and overall for extraction-style questions.
- `ROUGE-L` — range: [0, 1]
  - Longest Common Subsequence score measuring overlap between generated text and reference answers. Used for generation-level tasks like legal opinion drafting and summarization.

## Input / output format

**Input**: Task instructions and input query. For few-shot evaluation, a single task-specific demonstration example is prepended.

**Output**: Direct final answer, or intermediate chain-of-thought reasoning followed by the final response.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    f1 = f1_score(golds, predictions, average='macro')
    rouge_l = rouge_l_score(golds, predictions)
    return {'accuracy': acc, 'f1': f1, 'rouge_l': rouge_l}
```

## Common pitfalls

- Middle truncation is applied when inputs exceed context windows, potentially discarding critical information at the beginning or end of legal texts.
- Conflict detection tasks exhibit severe negative-class bias, causing many models to achieve 0.00 Y-F1 by predicting 'no conflict' for all cases.
- Few-shot demonstrations are explicitly drawn from a separate development set to prevent test-set leakage.
- Decoding temperature is fixed to 0 to eliminate sampling variance, which may penalize models that benefit from stochastic generation.

## Evidence (verbatim from paper)

> We adopt standardized evaluation metrics across tasks: Accuracy, F1 for multiple-choice and extraction-style questions, and ROUGE-L for generation-level tasks.

## Citation

```bibtex
@misc{nguyen2025vlegalbench,
  title={VLegal-Bench: Cognitively Grounded Benchmark for Vietnamese Legal Reasoning of Large Language Models},
  author={Nguyen Tien Dong et al.},
  year={2025},
  note={arXiv:2512.14554}
}
```

- arXiv: 2512.14554