Younger Eval

---
name: younger-eval
description: This benchmark evaluates graph neural networks and generative models on AI-generated neural network architectures represented as directed acyclic graphs. It probes two capabilities: local component-level refinement (predicting data flows and operator types) and global end-to-end architecture generation. Use when the user wants to benchmark on Younger, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.15132
  bibtex_key: yang2024younger
  confidence: high
---

# younger-eval

> Younger: The First Dataset for Artificial Intelligence-Generated Neural Network Architecture — Yang et al. (2024) (arXiv:2406.15132, 2024)

## What this evaluates

This benchmark evaluates graph neural networks and generative models on AI-generated neural network architectures represented as directed acyclic graphs. It probes two capabilities: local component-level refinement (predicting data flows and operator types) and global end-to-end architecture generation.

## Datasets

- **Younger** — total 7629; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used to evaluate multi-class operator design and data flow prediction tasks.
- `ACC` — range: [0, 1]
  - Accuracy, calculated as the ratio of correctly predicted instances to the total number of instances.
- `AUC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the model's ability to distinguish between classes.
- `AP` — range: [0, 1]
  - Average Precision, summarizing the precision-recall curve across different thresholds.
- `negative log-likelihood` — range: [0, inf)
  - Negative log-likelihood of the generated graph distribution on the test set, used to evaluate the global graph generation paradigm.

## Input / output format

**Input**: Directed acyclic graphs (DAGs) representing neural network architectures. Nodes contain discrete features: either just operator types ('Operator w/o Attributes') or operator types plus integer attribute configurations ('Operator w/ Attributes'). Graphs range from dozens to hundreds of thousands of nodes.

**Output**: For local paradigms: class labels for data flow design and operator types. For global paradigm: a generated directed acyclic graph with operator nodes and edges.

## Scoring recipe

```python
def compute_classification_metrics(y_true, y_pred):
    acc = (y_true == y_pred).mean()
    precision = precision_score(y_true, y_pred, average='macro')
    recall = recall_score(y_true, y_pred, average='macro')
    f1 = 2 * (precision * recall) / (precision + recall + 1e-8)
    return {'ACC': acc, 'Precision': precision, 'Recall': recall, 'F1': f1}

def compute_nll(model, test_graphs):
    log_probs = model.log_prob(test_graphs)
    return -log_probs.mean()
```

## Common pitfalls

- Treating architectures as heterogeneous graphs introduces unnecessary complexity; the paper explicitly evaluates them as homogeneous graphs where all nodes are treated as the same 'operator' type.
- Including operator attributes ('Operator w/ Attributes') drastically increases the feature space and degrades classification performance (F1/Precision/Recall) compared to using only operator types ('Operator w/o Attributes').
- The global generation paradigm is computationally constrained; models are only trained on architectures with 1-300 nodes, making evaluation on larger graphs out-of-distribution.

## Evidence (verbatim from paper)

> Despite the high accuracy achieved by all baselines, the F1 score, Precision, and Recall remain low. This is primarily attributed to the complex graph structures in Younger, which are characterized by many operator types.

## Citation

```bibtex
@misc{yang2024younger,
  title={Younger: The First Dataset for Artificial Intelligence-Generated Neural Network Architecture},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2406.15132}
}
```

- arXiv: 2406.15132