Streaming Evaluation

---
name: streaming-evaluation
description: "Ingest River-compatible streams, validate models online, handle
  delayed labels, and choose evaluation metrics."
disable-model-invocation: true
metadata:
  disco-role: operating
license: BSD 3-Clause
---

# River streaming evaluation

Use this sub-skill when the task is to turn a River-compatible data stream into online evaluation, delayed-label scoring, or checkpointed metric tracking.

## Load order

1. Read `references/data-streams.md` to normalize dataset shapes, adapters, target handling, and delayed-label streams.
2. Read `references/evaluation-and-metrics.md` before choosing metrics, progressive validation APIs, active learning behavior, or forecasting evaluation.
3. Read `references/troubleshooting.md` when stream parsing, moment/delay alignment, weights, metric compatibility, or optional adapters fail.
4. Use `scripts/stream_evaluation_smoke.py` for a tiny offline smoke test of CSV ingestion, delayed labels, and sample-weight routing.

## Route elsewhere

- Feature construction, encoders, target transforms, rolling feature engineering, and pipeline composition belong to `pipelines-and-features`.
- Base estimator contracts, cloning, tags, and check-estimator work belong to `online-core-api`.
- Supervised model family choice, optimizer/loss tuning, and model wrappers belong to `supervised-models`.
- Drift, anomaly, clustering, forecasting model design, active learning strategy design, bandits, recommenders, and other specialized model families belong to `specialized-workflows`.
- If the task is only about a particular model class or family, do not turn this into a stream-evaluation task.

## Fast routing cues

- Use `stream.iter_csv` for files, `stream.iter_array` for arrays, `stream.iter_frame` for eager dataframes, `stream.iter_libsvm` for sparse numeric text, and `stream.iter_sql` / `stream.iter_sklearn_dataset` when the source is already an SQLAlchemy query or scikit-learn `Bunch`.
- Use `Dataset.take(k)` to cap built-in or synthetic datasets when you need a bounded sample.
- Use `evaluate.progressive_val_score` for the standard predict-then-learn loop, `evaluate.iter_progressive_val_score` for checkpoints, and a manual loop only when you need custom control.
- Use `moment` and `delay` when labels arrive late or the stream should be replayed in arrival order.
- Use `simulate_qa` directly when you need to inspect the question/answer timeline before scoring.
- Use `metrics.Silhouette` for unlabeled clustering, `metrics.ROCAUC` / `metrics.RollingROCAUC` / `metrics.RollingPRAUC` for anomaly-style scores, and regression metrics such as `MAE`, `RMSE`, `R2`, or `SMAPE` for forecasting.