Time Series Transfer Eval

---
name: time_series_transfer_eval
description: Evaluates the effectiveness of transfer learning on time series data by comparing pre-trained models against models trained from scratch across intra-domain and cross-domain settings. It probes whether shared temporal structures enable knowledge transfer and how dataset size and domain similarity affect predictive performance and training convergence. Use when the user wants to benchmark on LEN-DB, SPEECH, EMG, S&P 500, LOMAX, STEAD, or asks about evaluating this task. Reports MAE, weighted F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2201.04449
  bibtex_key: otovic2022intra
  confidence: high
---

# time_series_transfer_eval

> Intra-domain and cross-domain transfer learning for time series data -- How transferable are the features? — Otović Erik et al. (arXiv:2201.04449, 2022)

## What this evaluates

Evaluates the effectiveness of transfer learning on time series data by comparing pre-trained models against models trained from scratch across intra-domain and cross-domain settings. It probes whether shared temporal structures enable knowledge transfer and how dataset size and domain similarity affect predictive performance and training convergence.

## Datasets

- **LEN-DB** — total ?; splits: train (-1), test (-1)
- **SPEECH** — total ?; splits: train (-1), test (-1)
- **EMG** — total ?; splits: train (-1), test (-1)
- **S&P 500** — total ?; splits: train (-1), test (-1)
- **LOMAX** — total ?; splits: train (-1), test (-1)
- **STEAD** — total ?; splits: train (-1), test (-1)

## Metrics

- `MAE` **(primary)** — range: other
  - Mean Absolute Error: the average of the absolute differences between predicted and actual values. Lower is better.
- `weighted F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, weighted by the number of true instances for each class. Higher is better.
- `convergence rate` — range: other
  - Number of training epochs or iterations required for the model to reach a predefined performance threshold or stabilize. Lower is better.

## Input / output format

**Input**: Time series sequences (optionally augmented with a 'stream max' input feature representing the maximum value in the sequence).

**Output**: Predicted continuous values for regression tasks, or class labels for classification tasks.

## Scoring recipe

```python
1. For each source-target-architecture triplet, train 70 models (10 LR multipliers × 7 reruns).
2. Select the best LR multiplier per run based on the primary metric.
3. Compute the average primary metric score across the 7 runs for both the TL model and the referent (scratch) model.
4. Calculate percentage difference: if MAE, diff = (ref - tl) / ref * 100; if F1, diff = (tl - ref) / ref * 100. Positive always indicates improvement.
5. Count wins (TL > ref) and losses across all triplets.
6. Apply a sign test to the win/loss counts for overall TL effectiveness.
7. For domain-pair analysis, compute the average percentage difference per pair, then apply a Wilcoxon signed-ranks test with Benjamini-Krieger-Yekutieli correction for multiple comparisons.
```

## Common pitfalls

- Misinterpreting the sign convention for percentage difference: positive values always denote improvement (decrease in MAE or increase in F1), regardless of the metric type.
- Skipping the grid search and 7-run averaging steps before statistical testing, which violates the paper's protocol for isolating optimal hyperparameters and reducing variance.
- Assuming negative transfer is permanent; the paper explicitly notes that negative transfer effects diminish as the target dataset size increases.

## Evidence (verbatim from paper)

> Table 2: Performance gains when using stream max. Positive difference in the case of LEN-DB and S&P 500 represents a decrease in MAE, while positive difference in the case of SPEECH and EMG represents an increase in the weighted F1 score.

## Citation

```bibtex
@misc{otovic2022intra,
  title={Intra-domain and cross-domain transfer learning for time series data -- How transferable are the features?},
  author={Otović Erik et al.},
  year={2022},
  note={arXiv:2201.04449}
}
```

- arXiv: 2201.04449