Vision Arch Gen Eval

---
name: vision-arch-gen-eval
description: Evaluates the classification performance of LLM-generated neural network architectures by training each for a single epoch on seven computer vision benchmarks. It measures Top-1 accuracy to assess architectural quality, while also tracking generation efficiency via hash validation speed and duplicate rejection rates. Use when the user wants to benchmark on MNIST, CelebA-Gender, CIFAR-10, CIFAR-100, ImageNette, SVHN, Places365, or asks about evaluating this task. Reports Top-1 accuracy after 1 epoch.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.24120
  bibtex_key: [REDACTED]
  confidence: high
---

# vision-arch-gen-eval

> Enhancing LLM-Based Neural Network Generation: Few-Shot Prompting and Efficient Validation for Automated Architecture Design — Vysyaraju et al. (2025) (arXiv:2512.24120, 2025)

## What this evaluates

Evaluates the classification performance of LLM-generated neural network architectures by training each for a single epoch on seven computer vision benchmarks. It measures Top-1 accuracy to assess architectural quality, while also tracking generation efficiency via hash validation speed and duplicate rejection rates.

## Datasets

- **MNIST** — total 70000; splits: full (70000)
- **CelebA-Gender** — total 202000; splits: full (202000)
- **CIFAR-10** — total 60000; splits: full (60000)
- **CIFAR-100** — total 60000; splits: full (60000)
- **ImageNette** — total 13000; splits: full (13000)
- **SVHN** — total 600000; splits: full (600000)
- **Places365** — total 1800000; splits: full (1800000)

## Metrics

- `Top-1 accuracy after 1 epoch` **(primary)** — range: [0, 1]
  - Fraction of correctly classified samples out of the total number of samples after training the generated architecture for exactly one epoch.
- `Hash computation time` — range: milliseconds
  - Wall-clock time required to compute the whitespace-normalized hash for deduplication validation.
- `Duplicates detected` — range: count
  - Count of functionally identical architectures identified and rejected during the validation phase.

## Input / output format

**Input**: Few-shot prompting examples (n=1 to 6) of neural network architectures alongside a target computer vision dataset specification.

**Output**: Generated neural network architecture code or configuration strings prefixed with 'alt-nn' variants.

## Scoring recipe

```python
for architecture in generated_architectures:
    for dataset in datasets:
        model = instantiate(architecture)
        train(model, dataset, epochs=1, optimizer='SGD', momentum=True)
        acc = compute_top1_accuracy(model, dataset)
        results[architecture][dataset] = acc

dataset_means = {ds: mean([r[ds] for r in results.values()]) for ds in datasets}
overall_mean = mean(dataset_means.values())
p_values = independent_t_test(variants, dataset_means)
```

## Common pitfalls

- Training is strictly limited to 1 epoch for rapid validation, which may not reflect the true convergence or final performance of the generated architectures.
- Batch sizes vary across datasets (64-4096), potentially introducing variance in accuracy that is unrelated to architectural quality.
- Whitespace-normalized hash validation only eliminates functionally identical architectures with formatting differences, missing semantically equivalent but structurally distinct designs.

## Evidence (verbatim from paper)

> Primary Metric: Top-1 accuracy after 1 epoch
Statistical Validation: Independent t-tests comparing variants within each dataset, dataset-balanced means for overall comparison

Training Configuration:
* •
    Epochs: 1 (for rapid validation)
* •
    Optimizer: SGD with momentum
* •
    Batch size: varies per dataset (typical: 64-4096)
* •
    Metric: Top-1 accuracy

## Citation

```bibtex
@misc{vysyaraju2025enhancing,
  title={Enhancing LLM-Based Neural Network Generation: Few-Shot Prompting and Efficient Validation for Automated Architecture Design},
  author={Vysyaraju et al. (2025)},
  year={2025},
  note={arXiv:2512.24120}
}
```

- arXiv: 2512.24120