Text To Code Customization Eval

---
name: text-to-code-customization-eval
description: Evaluates the ability of customized small language models to generate correct and domain-aligned Python code. It probes functional correctness on general programming tasks versus specialized library APIs (Scikit-learn, OpenCV) under different customization strategies like few-shot prompting, RAG, and LoRA fine-tuning. Use when the user wants to benchmark on HumanEval, BCSk, BCCV, or asks about evaluating this task. Reports Pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.16526
  bibtex_key: freire2026exploring
  confidence: high
---

# text-to-code-customization-eval

> Exploring different approaches to customize language models for domain-specific text-to-code generation — Luís Freire, Fernanda A. Andaló, Nicki Skafte Detlefsen (arXiv:2603.16526, 2026)

## What this evaluates

Evaluates the ability of customized small language models to generate correct and domain-aligned Python code. It probes functional correctness on general programming tasks versus specialized library APIs (Scikit-learn, OpenCV) under different customization strategies like few-shot prompting, RAG, and LoRA fine-tuning.

## Datasets

- **HumanEval** — total ?; splits: test (-1)
- **BCSk** — total ?; splits: test (-1)
- **BCCV** — total ?; splits: test (-1)

## Metrics

- `Pass@1` **(primary)** — range: [0, 1]
  - The fraction of problems for which the generated code passes all provided test cases on the first attempt.
- `Cosine Similarity (Sim.)` — range: [0, 1]
  - The cosine similarity between the embedding of the generated code and the reference solution, measuring stylistic and structural alignment with the target dataset.

## Input / output format

**Input**: Natural language task description, optionally augmented with few-shot examples or dynamically retrieved context snippets.

**Output**: A Python code snippet implementing the requested function or task.

## Scoring recipe

```python
def compute_pass_at_1(generated_code, test_cases):
    return 1.0 if run_tests(generated_code, test_cases) else 0.0

def compute_cosine_similarity(gen_code, ref_code):
    gen_emb = embed(gen_code)
    ref_emb = embed(ref_code)
    return cosine_similarity(gen_emb, ref_emb)
```

## Common pitfalls

- RAG consistently improves similarity scores but does not guarantee higher Pass@1, as retrieved examples may introduce unnecessary operations.
- Few-shot prompting performance is highly sensitive to context window limits and can degrade with too many examples.
- Cosine similarity correlates with benchmark accuracy but measures stylistic alignment rather than functional correctness.

## Evidence (verbatim from paper)

> Performance is measured using benchmark-based metrics (Pass@1) and similarity metrics capturing alignment with the target datasets. For general Python tasks we use the HumanEval benchmark, while for library-specific tasks we use BigBenchCode subsets: BCSk for Scikit-learn and BCCV for OpenCV. Pass@1 measures benchmark correctness, and Sim. denotes cosine similarity to reference solutions.

## Citation

```bibtex
@misc{freire2026exploring,
  title={Exploring different approaches to customize language models for domain-specific text-to-code generation},
  author={Luís Freire, Fernanda A. Andaló, Nicki Skafte Detlefsen},
  year={2026},
  note={arXiv:2603.16526}
}
```

- arXiv: 2603.16526