Multipl E Low Resource Eval

---
name: multipl-e-low-resource-eval
description: Evaluates large language models' ability to generate functionally correct code in low-resource programming languages (R and Racket). It probes how well in-context learning strategies and fine-tuning adapt pre-trained models to languages with limited training data and documentation. Use when the user wants to benchmark on MultiPL-E, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.19085
  bibtex_key: giagnorio2025enhancing
  confidence: high
---

# multipl-e-low-resource-eval

> Enhancing Code Generation for Low-Resource Languages: No Silver Bullet — Giagnorio et al. (2025) (arXiv:2501.19085, 2025)

## What this evaluates

Evaluates large language models' ability to generate functionally correct code in low-resource programming languages (R and Racket). It probes how well in-context learning strategies and fine-tuning adapt pre-trained models to languages with limited training data and documentation.

## Datasets

- **MultiPL-E** — total 161; splits: test (161)

## Metrics

- `pass@1` **(primary)** — range: [0, 1]
  - The probability that at least one of the n=50 generated code samples is functionally correct. Computed as 1 - (C(n-c, k) / C(n, k)) where c is the number of correct samples, n=50, and k=1. In practice, it scores 1.0 if any of the 50 samples passes all unit tests, else 0.0.

## Input / output format

**Input**: Function docstring and signature, optionally augmented with in-context examples (Python-to-target translations, translation rules, or few-shot target-language examples) depending on the technique.

**Output**: Function body implementation (concrete code) in the target low-resource language (R or Racket).

## Scoring recipe

```python
def compute_pass_at_1(predictions, test_cases):
    # predictions: list of 50 generated code strings
    # test_cases: list of unit tests for the function
    passed = [run_tests(code, test_cases) for code in predictions]
    # pass@1 = 1 if at least one sample passes, else 0
    return 1.0 if any(passed) else 0.0
```

## Common pitfalls

- Fine-tuned models are evaluated by reporting only the best epoch's results, not the final epoch, to avoid performance degradation from overfitting on small datasets.
- pass@1 is computed over 50 sampled outputs per program, but the metric only requires one successful sample to score 1.0, which can mask partial correctness or inconsistent generation.
- The 161 programs are specific to R and Racket subsets; they do not represent the full MultiPL-E benchmark which covers many other languages.

## Evidence (verbatim from paper)

> As in our previous study, our evaluation metric is the pass@1 rate with n = 50 repetitions. In this case, instead of performance across languages, we compare performance across techniques.

## Citation

```bibtex
@misc{giagnorio2025enhancing,
  title={Enhancing Code Generation for Low-Resource Languages: No Silver Bullet},
  author={Giagnorio et al. (2025)},
  year={2025},
  note={arXiv:2501.19085}
}
```

- arXiv: 2501.19085