Tsmd F1 Eval

---
name: tsmd-f1-eval
description: Evaluates the quality of discovered motif sets in time series by measuring alignment with ground truth segments. It accounts for variable-length patterns and time warping while penalizing both false discoveries and missed patterns. Use when the user wants to benchmark on TSMD benchmark datasets, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.17582
  bibtex_key: vanwesenbeeck2023locomotif
  confidence: high
---

# tsmd-f1-eval

> LoCoMotif: Discovering time-warped motifs in time series — Van Wesenbeeck et al. (2023) (arXiv:2311.17582, 2023)

## What this evaluates

Evaluates the quality of discovered motif sets in time series by measuring alignment with ground truth segments. It accounts for variable-length patterns and time warping while penalizing both false discoveries and missed patterns.

## Datasets

- **TSMD benchmark datasets** — total ?; splits: (unstated)

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Precision and recall are computed via a matching matrix between ground truth and discovered motif sets. Segments are matched if their Jaccard similarity exceeds 0.5. Precision is the ratio of correctly matched segments to all predicted segments; recall is the ratio to all ground truth segments. F1 is the harmonic mean of precision and recall.

## Input / output format

**Input**: Ground truth motif sets G and discovered motif sets M, each comprising non-overlapping time series segments.

**Output**: Scalar F1-score value between 0 and 1.

## Scoring recipe

```python
# Match segments with Jaccard > 0.5
matches = greedy_match(gold_segments, pred_segments, threshold=0.5)
# Build matching matrix M* of size (k'+1) x (k+1)
M_star = build_matching_matrix(gold_sets, pred_sets, matches)
k_min = min(len(gold_sets), len(pred_sets))
tp = sum(M_star[i, i] for i in range(k_min))
pred_total = sum(M_star[i, j] for j in range(k_min) for i in range(len(gold_sets)+1))
gold_total = sum(M_star[i, j] for i in range(len(gold_sets)) for j in range(len(pred_sets)+1))
precision = tp / pred_total if pred_total > 0 else 0
recall = tp / gold_total if gold_total > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return f1
```

## Common pitfalls

- Prior metrics often only support single motif sets or fixed lengths, failing to capture variable-length patterns.
- Many existing metrics do not penalize false discoveries, leading to inflated quality scores.
- The segment matching threshold is strictly set to a Jaccard similarity of 0.5; using a different threshold alters the matching matrix and final score.

## Evidence (verbatim from paper)

> To define F1-score, we use the micro-average of precision and recall for multi-class classification (Grandini et al. 2020): $$ \mathrm {p r e c i s i o n} = \frac {\sum_ {i = 1} ^ {\kappa_ {\mathrm {m i n}}} M _ {i , i} ^ {*}}{\sum_ {j = 1} ^ {\kappa_ {\mathrm {m i n}}} \left(\sum_ {i = 1} ^ {\kappa^ {\prime} + 1} M _ {i , j} ^ {*}\right)} \quad \mathrm {a n d} \quad \mathrm {r e c a l l} = \frac {\sum_ {i = 1} ^ {\kappa_ {\mathrm {m i n}}} M _ {i , i} ^ {*}}{\sum_ {i = 1} ^ {\kappa^ {\prime}} \left(\sum_ {j = 1} ^ {\kappa + 1} M _ {i , j} ^ {*}\right)} $$ where $\kappa_{\mathrm{min}} = \min (\kappa ,\kappa^{\prime})$ (see Fig. 6b for an example). Our F1-score is defined as the harmonic mean of this precision and recall, and evaluates the quality of $\mathcal{M}$ in all aspects, as precision penalizes unmatched segments in discovered motif sets (that is, false discoveries) and recall penalizes undiscovered segments in GT motif sets.

## Citation

```bibtex
@misc{vanwesenbeeck2023locomotif,
  title={LoCoMotif: Discovering time-warped motifs in time series},
  author={Van Wesenbeeck et al. (2023)},
  year={2023},
  note={arXiv:2311.17582}
}
```

- arXiv: 2311.17582