Openml Cc18 Eval

---
name: openml-cc18-eval
description: Evaluates machine learning classifiers on a curated collection of standardized classification tasks. It probes the reproducibility and comparability of algorithm performance across diverse datasets under consistent, machine-readable evaluation protocols. Use when the user wants to benchmark on OpenML-CC18, or asks about evaluating this task. Reports accuracy_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1708.03731
  bibtex_key: [REDACTED]
  confidence: high
---

# openml-cc18-eval

> OpenML Benchmarking Suites — Bischl et al. (2017) (arXiv:1708.03731, 2017)

## What this evaluates

Evaluates machine learning classifiers on a curated collection of standardized classification tasks. It probes the reproducibility and comparability of algorithm performance across diverse datasets under consistent, machine-readable evaluation protocols.

## Datasets

- **OpenML-CC18** — total ?; splits: train (-1), test (-1); repo https://github.com/openml/benchmark-suites

## Metrics

- `accuracy_score` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly predicted instances out of the total number of instances in the evaluation set.

## Input / output format

**Input**: An OpenML task object containing a dataset, predefined train/test splits, and evaluation instructions, along with a machine learning learner or pipeline configuration.

**Output**: A run object containing the computed metric score (accuracy) for the task, which can be published to the OpenML platform for sharing and comparison.

## Scoring recipe

```python
benchmark_suite = study.get_suite('OpenML-CC18')
for task_id in benchmark_suite.tasks:
    task = tasks.get_task(task_id)
    run = runs.run_model_on_task(clf, task)
    score = run.get_metric_fn(metrics.accuracy_score)
    print(f'Accuracy: {score.mean():.2f}')
```

## Common pitfalls

- Confusing the general OpenML dataset repository with the curated OpenML-CC18 benchmarking suite.
- Assuming static train/test splits; OpenML tasks define splits programmatically, so results depend on the exact task configuration.
- Overlooking that the suite is dynamic and can be extended or corrected over time, meaning benchmarks are not permanent snapshots.

## Evidence (verbatim from paper)

> An OpenML benchmarking suite is a set of OpenML tasks carefully selected to evaluate algorithms under a precise set of conditions.
score = run.get_metric_fn(metrics.accuracy_score)  # compute and print the accuracy score

## Citation

```bibtex
@misc{bischl2017openml,
  title={OpenML Benchmarking Suites},
  author={Bischl et al. (2017)},
  year={2017},
  note={arXiv:1708.03731}
}
```

- arXiv: 1708.03731