Vibepass Eval

---
name: vibepass-eval
description: Evaluates LLMs on fault-targeted test generation and fault-targeted program repair. It probes discriminative fault detection, fault hypothesis generation, and the ability to debug subtle semantic bugs under diagnostic guidance. Use when the user wants to benchmark on VIBEPASS, or asks about evaluating this task. Reports D_{IO}.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.15921
  bibtex_key: bansal2026vibepass
  confidence: high
---

# vibepass-eval

> VIBEPASS: Can Vibe Coders Really Pass the Vibe Check? — Bansal et al. (2026) (arXiv:2603.15921, 2026)

## What this evaluates

Evaluates LLMs on fault-targeted test generation and fault-targeted program repair. It probes discriminative fault detection, fault hypothesis generation, and the ability to debug subtle semantic bugs under diagnostic guidance.

## Datasets

- **VIBEPASS** — total ?; splits: test (-1)

## Metrics

- `D_{IO}` **(primary)** — range: percent
  - Discriminative input/output validity. Measures whether a generated test case has valid inputs and outputs AND successfully triggers a fault in the buggy code.
- `J+D_{IO}` — range: percent
  - Joint metric requiring both correct bug judgment (J) and discriminative test generation (D_{IO}).
- `SR` — range: percent
  - Success Rate for fault-targeted program repair, measuring whether the repaired code passes all validation checks.
- `P@1` — range: percent
  - Pass@1 for code generation or repair, indicating the proportion of instances where the first generated solution is correct.

## Input / output format

**Input**: Buggy code snippets with problem descriptions. For test generation: optionally a bug label (Bug-Aware) or none (Bug-Discovery). For repair: buggy code, problem description, and optionally a test case (NoTest, IntTest, ExtTest).

**Output**: Generated test cases (inputs and expected outputs), a binary judgment on whether the code is buggy, and/or a repaired version of the code.

## Scoring recipe

```python
def score_dio(generated_test, buggy_code):
    valid_io = check_input_output_validity(generated_test)
    triggers_fault = run_test_on_code(generated_test, buggy_code)
    return valid_io and triggers_fault

def score_sr(repaired_code, test_suite):
    return all(run_test_on_code(test, repaired_code) for test in test_suite)
```

## Common pitfalls

- Confusing syntactic input validity (V_I) with discriminative fault-triggering capability (D_I), as models often generate format-compliant tests that fail to expose bugs.
- Assuming external test guidance always improves debugging performance; the evaluation shows self-generated tests or no-test baselines often outperform externally provided tests.
- Treating raw code generation ability (P@1) as a reliable proxy for fault-targeted program repair success.

## Evidence (verbatim from paper)

> In the Bug-Aware, the model knows the code is buggy, and performance measures FT-Test generation quality directly:  $V_I / V_{IO}$  for input/output validity, and  $D_I / D_{IO}$  for discriminative effectiveness. In the Bug-Discovery, J denotes judgment accuracy (buggy or not), and joint metrics  $(J^{+*})$  require both correct judgment and the corresponding quality criterion.

## Citation

```bibtex
@misc{bansal2026vibepass,
  title={VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?},
  author={Bansal et al. (2026)},
  year={2026},
  note={arXiv:2603.15921}
}
```

- arXiv: 2603.15921