Consistencychecker Eval

---
name: consistencychecker-eval
description: Evaluates LLM generalization and functional consistency by measuring how well models preserve core functionality after iterative, reversible transformations. It probes cumulative error and path-specific divergence across multi-step transformation sequences without relying on static benchmarks. Use when the user wants to benchmark on ConsistencyChecker (Dynamic), or asks about evaluating this task. Reports forest-level consistency score (C3(F)).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.12376
  bibtex_key: [REDACTED]
  confidence: high
---

# consistencychecker-eval

> ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities — Hong et al. (2025) (arXiv:2506.12376, 2025)

## What this evaluates

Evaluates LLM generalization and functional consistency by measuring how well models preserve core functionality after iterative, reversible transformations. It probes cumulative error and path-specific divergence across multi-step transformation sequences without relying on static benchmarks.

## Datasets

- **ConsistencyChecker (Dynamic)** — total ?; splits: (unstated); repo https://github.com/ulab-uiuc/consistencychecker

## Metrics

- `forest-level consistency score (C3(F))` **(primary)** — range: [0, 1]
  - Average of path-level consistency scores across all trees in a forest at depth n=3. Path consistency C(P) is the similarity (e.g., cosine similarity or BLEU) between the initial node output and the final node output after n reversible transformations. C3(F) = (1/M) * sum_{m=1 to M} C_3(T_m).

## Input / output format

**Input**: A root node containing an initial text or code instance, a set of test inputs, and a sequence of reversible transformation operations (e.g., translate-then-translate-back, add-logging-then-remove-logging).

**Output**: The model generates transformed versions of the input at each tree depth. The evaluation computes execution outputs or semantic embeddings for each node to calculate similarity scores.

## Scoring recipe

```python
def compute_c3_forest(forest_trees, depth=3):
    forest_scores = []
    for tree in forest_trees:
        paths = get_all_paths(tree, depth)
        path_scores = []
        for path in paths:
            initial_out = exec_or_embed(path[0].content)
            final_out = exec_or_embed(path[-1].content)
            path_scores.append(similarity(initial_out, final_out))
        tree_score = mean(path_scores)
        forest_scores.append(tree_score)
    return mean(forest_scores)
```

## Common pitfalls

- Assuming the evaluation relies on static external benchmarks; the framework is benchmark-free and dynamically generates transformation trees.
- Treating the similarity metric as fixed; the paper explicitly allows cosine similarity or BLEU, which produce different absolute scales.
- Varying the depth n arbitrarily; the final reported metric strictly uses n=3 to balance computational cost and error accumulation.

## Evidence (verbatim from paper)

> To balance these trade-offs, we select n=3 and use the forest-level consistency score C3(F) as the final metric in our ConsistencyChecker framework. C_n(F) := (1/M) \sum_{m=1}^{M} C_n(T_m), where C_n(T) denotes the consistency at path length n of tree T.

## Citation

```bibtex
@misc{hong2025consistencychecker,
  title={ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities},
  author={Hong et al. (2025)},
  year={2025},
  note={arXiv:2506.12376}
}
```

- arXiv: 2506.12376