Gru D Eval

---
name: gru-d-eval
description: This evaluation probes a model's ability to handle multivariate time series with missing values by jointly learning temporal dependencies and informative missing patterns. It tests classification performance on clinical and synthetic datasets, measuring how well the model exploits masking and time-interval information for early prediction and multi-task diagnosis. Use when the user wants to benchmark on Gesture, PhysioNet Challenge 2012, MIMIC-III, or asks about evaluating this task. Reports AUC score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1606.01865
  bibtex_key: che2016recurrent
  confidence: high
---

# gru-d-eval

> Recurrent Neural Networks for Multivariate Time Series with Missing Values — Che et al. (2016) (arXiv:1606.01865, 2016)

## What this evaluates

This evaluation probes a model's ability to handle multivariate time series with missing values by jointly learning temporal dependencies and informative missing patterns. It tests classification performance on clinical and synthetic datasets, measuring how well the model exploits masking and time-interval information for early prediction and multi-task diagnosis.

## Datasets

- **Gesture** — total 378; splits: train (-1), val (-1), test (-1)
- **PhysioNet Challenge 2012** — total 8000; splits: train (-1), val (-1), test (-1)
- **MIMIC-III** — total 19714; splits: train (-1), val (-1), test (-1)

## Metrics

- `AUC score` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic (ROC) curve. It measures the probability that a classifier ranks a randomly chosen positive instance higher than a randomly chosen negative one, averaged across all classification thresholds.

## Input / output format

**Input**: Multivariate time series sequences of measurements, accompanied by binary masking vectors (indicating observed vs. missing entries) and time-interval vectors (encoding gaps between observations). Inputs are normalized to zero mean and unit variance. Non-RNN baselines use fixed-length, regularly sampled sequences with forward/backward imputation.

**Output**: Classification probabilities or hard labels via a soft-max regressor applied to the final hidden state. Supports binary classification (e.g., mortality) and multi-task classification (e.g., 4 PhysioNet tasks or 20 MIMIC-III ICD-9 codes).

## Scoring recipe

```python
auc_scores = []
for fold in range(5):
    train, val, test = split_into_folds(dataset, k=5)
    model = train_model(train, val)  # early stopping on val
    preds = model.predict(test)
    auc = compute_auc(y_true=test.labels, y_score=preds)
    auc_scores.append(auc)
final_score = np.mean(auc_scores)
report_mean_std(final_score, auc_scores)
```

## Common pitfalls

- Non-RNN baselines require fixed-length inputs and forward/backward imputation, which can distort temporal dynamics compared to RNNs that handle variable lengths natively.
- Ignoring informative missingness (masking and time intervals) leads to suboptimal performance, especially when missingness patterns correlate with the target label.
- Early stopping is applied on a validation set, but the exact validation split strategy beyond 5-fold CV is not detailed, potentially affecting reproducibility.

## Evidence (verbatim from paper)

> We report the results from 5-fold cross validation in terms of area under the ROC curve (AUC score).

## Citation

```bibtex
@misc{che2016recurrent,
  title={Recurrent Neural Networks for Multivariate Time Series with Missing Values},
  author={Che et al. (2016)},
  year={2016},
  note={arXiv:1606.01865}
}
```

- arXiv: 1606.01865