Multilogbench Eval

---
name: multilogbench-eval
description: Evaluates LLMs' ability to generate appropriate logging statements for code callables across six programming languages, testing both snapshot-based code understanding and revision-history-based code evolution contexts. Use when the user wants to benchmark on MultiLogBench, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.17529
  bibtex_key: [REDACTED]
  confidence: medium
---

# multilogbench-eval

> Single-Language Evidence Is Insufficient for Automated Logging: A Multilingual Benchmark and Empirical Study with LLMs — Zhong et al. (2026) (arXiv:2604.17529, 2026)

## What this evaluates

Evaluates LLMs' ability to generate appropriate logging statements for code callables across six programming languages, testing both snapshot-based code understanding and revision-history-based code evolution contexts.

## Datasets

- **MultiLogBench** — total ?; splits: repository-snapshot (-1), revision-history (-1), transformed (-1); repo https://github.com/logresearch/MultiLogBench

## Metrics

- `exact-match accuracy` **(primary)** — range: [0, 1]
  - Percentage of generated logging statements that exactly match the developer-authored gold statement.

## Input / output format

**Input**: A callable (function/method) extracted from a frozen repository snapshot or a parent revision, with the target logging statement removed and formatting gaps repaired.

**Output**: A single logging statement string to be inserted at the target site.

## Scoring recipe

```python
def score(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip() == gold.strip():
            correct += 1
    return correct / len(golds)
```

## Common pitfalls

- Confusing repository-snapshot data (pre-existing logs) with revision-history data (log-introducing edits)
- Failing to account for framework-specific logging APIs vs. standard ones
- Overestimating performance due to pretraining contamination on GitHub-sourced code

## Evidence (verbatim from paper)

> Repository-snapshot data serves as the primary evaluation source. Its instances are mined from a frozen snapshot of each selected repository, and each target logging statement is a developer-authored statement that already exists in that revision.

## Citation

```bibtex
@misc{zhong2026multilogbench,
  title={Single-Language Evidence Is Insufficient for Automated Logging: A Multilingual Benchmark and Empirical Study with LLMs},
  author={Zhong et al. (2026)},
  year={2026},
  note={arXiv:2604.17529}
}
```

- arXiv: 2604.17529