Evaluation And Data

---
name: evaluation-and-data
description: "Router for imbalanced-learn dataset fabrication, benchmark
  loading, imbalance-aware metrics, and categorical distance tools."
metadata:
  disco-role: operating
disable-model-invocation: true
license: MIT
---

# evaluation-and-data

Use this sub-skill when the task is about preparing imbalanced datasets or
measuring model quality on imbalanced classification problems.

This sub-skill owns the package surfaces that shape or evaluate data:

- `make_imbalance`
- `fetch_datasets`
- `classification_report_imbalanced`
- `geometric_mean_score`
- `sensitivity_specificity_support`
- `sensitivity_score`
- `specificity_score`
- `make_index_balanced_accuracy`
- `macro_averaged_mean_absolute_error`
- `ValueDifferenceMetric`

It also owns the common-pitfalls guidance around data leakage and evaluation
order.

## What to do first

1. Decide whether the user wants to create an imbalanced dataset, load a cached
   benchmark dataset, or evaluate predictions.
2. If the task is about evaluation, decide whether the labels are binary,
   multiclass, or ordinal.
3. If the task is about categorical similarity, check whether the inputs are
   categorical-only and need `ValueDifferenceMetric`.
4. Keep any resampling inside the training branch; do not evaluate on a fully
   resampled dataset unless that is explicitly the point of the example.
5. Verify the output on a tiny dataset before applying the same pattern to a
   real one.

## Typical routing cues

- `make_imbalance`, `fetch_datasets`
- `classification_report_imbalanced`, `geometric_mean_score`
- `sensitivity_score`, `specificity_score`, `macro_averaged_mean_absolute_error`
- `ValueDifferenceMetric`
- `common_pitfalls`, leakage, balanced accuracy, pandas output

## When to read the bundled references

- `references/workflows.md` for dataset shaping and metric selection.
- `references/api-reference.md` for the compact dataset and metric catalog.
- `references/troubleshooting.md` when downloading, label encoding, or metric
  selection fails.

## Common choices

- Use `make_imbalance` when the task starts from a balanced dataset and needs a
  deliberate class-skew scenario.
- Use `fetch_datasets` when the task needs one of the curated benchmark
  datasets and offline cache behavior is acceptable.
- Use `classification_report_imbalanced` when the user wants a human-readable
  per-class summary.
- Use `geometric_mean_score` when class-wise balance matters more than overall
  accuracy.
- Use `sensitivity_*` and `specificity_*` when the domain is medical or recall-
  versus-specificity oriented.
- Use `macro_averaged_mean_absolute_error` for ordinal imbalance problems.
- Use `ValueDifferenceMetric` when the task is about categorical pairwise
  distances.

## Native evidence to keep in mind

These repo tests are the most relevant later verification anchors for this
sub-skill:

- `imblearn/datasets/tests/test_imbalance.py::test_make_imbalanced_iris`
- `imblearn/datasets/tests/test_zenodo.py::test_fetch`
- `imblearn/metrics/tests/test_classification.py::test_classification_report_imbalanced_multiclass`
- `imblearn/metrics/tests/test_classification.py::test_geometric_mean_multiclass`
- `imblearn/metrics/tests/test_classification.py::test_macro_averaged_mean_absolute_error`
- `imblearn/metrics/tests/test_pairwise.py::test_value_difference_metric`
- `imblearn/tests/test_common.py::test_pandas_column_name_consistency`

## Package-specific cautions

- `fetch_datasets` is a network/cache helper, not a pure offline primitive.
- Metric results depend on label order, averaging choice, and the selected
  positive label.
- `ValueDifferenceMetric` expects categorical values to be encoded in a way the
  metric can interpret.
- Leakage-safe evaluation means splitting before resampling.

## Use the scripts

- `scripts/dataset_imbalance_demo.py` for tiny `make_imbalance` and cached
  dataset checks.
- `scripts/metrics_report_demo.py` for compact metric output on fixed labels.
- `scripts/vdm_demo.py` for categorical distance checks.