M Design Model Selection Eval

---
name: m-design-model-selection-eval
description: Evaluates the effectiveness and refinement efficiency of neural network architecture search and selection methods on graph datasets. It measures how well a method can find near-optimal models within a limited search budget and how quickly it reaches a target performance level across diverse graph topologies and tasks. Use when the user wants to benchmark on Graph Architecture Search Benchmark (22 datasets), or asks about evaluating this task. Reports classification accuracy / AUC-ROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.15336
  bibtex_key: wang2025mdesign
  confidence: high
---

# m-design-model-selection-eval

> Beyond Model Base Selection: Weaving Knowledge to Master Fine-grained Neural Network Design — Wang et al. (2025) (arXiv:2507.15336, 2025)

## What this evaluates

Evaluates the effectiveness and refinement efficiency of neural network architecture search and selection methods on graph datasets. It measures how well a method can find near-optimal models within a limited search budget and how quickly it reaches a target performance level across diverse graph topologies and tasks.

## Datasets

- **Graph Architecture Search Benchmark (22 datasets)** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `classification accuracy / AUC-ROC` **(primary)** — range: [0, 1]
  - Proportion of correctly classified instances (accuracy) or area under the receiver operating characteristic curve (AUC-ROC), depending on the task type. Evaluated under a maximum search budget of 100 model-testings.
- `number of model-testings` — range: integer
  - The count of model evaluations required to reach a predefined target performance level. Capped at 100 if the target is not met within the search budget.
- `Kendall rank correlation` — range: [-1, 1]
  - Measures the ordinal association between predicted and actual local modification consistency rankings, used in ablation studies.

## Input / output format

**Input**: Graph datasets with node features, edges, and labels, along with a predefined model design space and task metadata.

**Output**: Selected or refined neural network architecture, and its performance metrics (accuracy/AUC-ROC) after up to 100 model-testings.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, target_perf, max_budget=100):
    accuracy = sum(1 for p, g in zip(predictions, gold_labels) if p == g) / len(gold_labels)
    iterations = max_budget
    for i, perf in enumerate(predictions):
        if perf >= target_perf:
            iterations = i + 1
            break
    return {'accuracy': accuracy, 'model_testings': iterations}
```

## Common pitfalls

- Search budget is strictly capped at 100 model-testings; methods failing to reach the target performance within this limit are recorded as 100.
- Target performance is task-specific, defined as the halfway point of M-DESIGN's 50-iteration run, not a fixed global threshold.
- All results are averaged over 10 repeated trials; single-run evaluations will deviate significantly from reported numbers.

## Evidence (verbatim from paper)

> We evaluate the model selection (1) effectiveness with the classification accuracy or AUC-ROC on all data-task pairs under a maximum search budget of 100 model-testings and (2) refinement efficiency with the number of model-testings (i.e., the basic time unit in the benchmarking setting) required for methods to achieve the target performance level, i.e., the halfway performance marked by running 50 iterations of M-DESIGN on the node classification task.

## Citation

```bibtex
@misc{wang2025mdesign,
  title={Beyond Model Base Selection: Weaving Knowledge to Master Fine-grained Neural Network Design},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2507.15336}
}
```

- arXiv: 2507.15336