Scratheval Eval

---
name: scratheval-eval
description: Evaluates LLMs' ability to understand, diagnose, and repair bugs in multimodal, event-driven block-based programming environments (Scratch). It probes functional correctness, structured bug explanation, trigger/mechanism identification, and patch minimality/semantic preservation. Use when the user wants to benchmark on ScratchEval, or asks about evaluating this task. Reports G-Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.00757
  bibtex_key: si2026scratheval
  confidence: high
---

# scratheval-eval

> ScratchEval : A Multimodal Evaluation Framework for LLMs in Block-Based Programming — Si et al. (2026) (arXiv:2602.00757, 2026)

## What this evaluates

Evaluates LLMs' ability to understand, diagnose, and repair bugs in multimodal, event-driven block-based programming environments (Scratch). It probes functional correctness, structured bug explanation, trigger/mechanism identification, and patch minimality/semantic preservation.

## Datasets

- **ScratchEval** — total 100; splits: test (100)

## Metrics

- `G-Acc` **(primary)** — range: [0, 1]
  - Accuracy of recovering the project-level intent and functional structure (e.g., project goal, main loop, core sprites).
- `U-Acc` — range: [0, 1]
  - Strict joint accuracy where the model's structured explanation correctly identifies the trigger, mechanism, and outcome matching the reference.
- `T-F1` — range: [0, 1]
  - F1-score for identifying the specific event or action that initiates faulty behavior against gold-standard triggers.
- `M-Acc` — range: [0, 1]
  - Accuracy of classifying the underlying causal bug mechanism (e.g., concurrency/timing, state initialization, missing handlers).
- `Fix Success Rate` — range: [0, 1]
  - Percentage of bugs where the model's generated patch passes all executable VM-level test suites.

## Input / output format

**Input**: Buggy Scratch project exported as .sb3 (containing blocks, sprites, and metadata) paired with original multimedia assets (PNG/SVG images, WAV sounds). No gameplay videos are provided.

**Output**: Structured bug explanation (trigger, mechanism, outcome) and/or a repaired project patch (modified blocks/sprites). For repair tasks, the output is evaluated as a complete .sb3 project.

## Scoring recipe

```python
def score_scratheval(predictions, gold):
    g_acc = 1.0 if predictions.project_intent == gold.project_intent else 0.0
    u_acc = 1.0 if (predictions.trigger == gold.trigger and
                    predictions.mechanism == gold.mechanism and
                    predictions.outcome == gold.outcome) else 0.0
    t_f1 = compute_f1(gold.triggers, predictions.triggers)
    m_acc = 1.0 if predictions.mechanism_tag == gold.mechanism_tag else 0.0
    fix_success = 1.0 if execute_vm_tests(predictions.patch) == 'PASS' else 0.0
    return {'G-Acc': g_acc, 'U-Acc': u_acc, 'T-F1': t_f1, 'M-Acc': m_acc, 'Fix Success': fix_success}
```

## Common pitfalls

- Models frequently over-edit or introduce auxiliary state/logic beyond the localized fix, violating minimality.
- Patches often exhibit semantic drift, passing tests but altering unintended project behaviors compared to the original intent.
- Models apply sequential-code heuristics that fail to respect Scratch’s event-driven concurrency and broadcast ordering.

## Evidence (verbatim from paper)

> Global Understanding (G-Acc) = accuracy of recovering the project-level intent/structure; Bug Understanding (U-Acc) = percentage of bugs where the model’s structured explanation matches the reference (trigger, mechanism, and outcome); T-F1 = trigger identification F1-score; M-Acc = mechanism tag accuracy; Fix Gains = number of previously failing repairs that became correct after LoRA tuning (i.e., fail→success relative to Qwen zero-shot on the 100-bug evaluation set).

## Citation

```bibtex
@misc{si2026scratheval,
  title={ScratchEval : A Multimodal Evaluation Framework for LLMs in Block-Based Programming},
  author={Si et al. (2026)},
  year={2026},
  note={arXiv:2602.00757}
}
```

- arXiv: 2602.00757