Nl2sh Eval

---
name: nl2sh-eval
description: This benchmark evaluates the ability of large language models to translate natural language instructions into executable Bash commands. It probes functional correctness by comparing model-generated commands against ground-truth commands using a functional equivalence heuristic that combines command execution with LLM-based output analysis. Use when the user wants to benchmark on NL2SH, InterCode-ALFA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.06858
  bibtex_key: westenfelder2025llmsupported
  confidence: high
---

# nl2sh-eval

> LLM-Supported Natural Language to Bash Translation — Westenfelder et al. (2025) (arXiv:2502.06858, 2025)

## What this evaluates

This benchmark evaluates the ability of large language models to translate natural language instructions into executable Bash commands. It probes functional correctness by comparing model-generated commands against ground-truth commands using a functional equivalence heuristic that combines command execution with LLM-based output analysis.

## Datasets

- **NL2SH** — total 41539; splits: train (40939), test (600); repo https://github.com/westenfelder/NL2SH
- **InterCode-ALFA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly translated commands. Correctness is determined by the exec + mxbai-embed functional equivalence heuristic (FEH), which executes both the ground-truth and predicted commands, compares their stdout via cosine similarity of embeddings, and uses an LLM to verify functional equivalence with a 0.75 threshold.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall for the functional equivalence heuristic evaluation, measuring how well the heuristic distinguishes equivalent from non-equivalent command pairs.

## Input / output format

**Input**: Natural language instruction describing a shell task. For FEH evaluation, pairs of Bash commands (ground truth and predicted or rotated).

**Output**: A single Bash command string.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        out_pred = execute_command(pred)
        out_gold = execute_command(gold)
        sim = cosine_similarity(embed(out_pred), embed(out_gold))
        llm_ok = llm_evaluate_functional_equivalence(pred, gold, out_pred, out_gold)
        if sim >= 0.75 and llm_ok:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- The non-equivalent test pairs are created by arbitrarily rotating the third column of the dataset, which may not reflect realistic functional divergence or command syntax errors.
- All LLM evaluations use temperature=0 and seed=123, which eliminates stochasticity and may overestimate consistency compared to real-world usage.
- The 0.75 threshold for cosine similarity and LLM verdicts is fixed across all heuristics, potentially biasing precision/recall trade-offs for different model families.

## Evidence (verbatim from paper)

> All models are evaluated using version 0.3.6 of the InterCode-ALFA benchmark with the execution + mxbai-embed FEH. Accuracy is measured using the exec + mxbai-embed FEH.

## Citation

```bibtex
@misc{westenfelder2025llmsupported,
  title={LLM-Supported Natural Language to Bash Translation},
  author={Westenfelder et al. (2025)},
  year={2025},
  note={arXiv:2502.06858}
}
```

- arXiv: 2502.06858