Weak Annotation Har Eval

---
name: weak-annotation-har-eval
description: Evaluates inertial-based activity recognition models trained on weakly-supervised labels generated via vision foundation model clustering, benchmarked against fully-supervised and few-shot baselines. Use when the user wants to benchmark on WEAR, Wetlab, ActionSense, or asks about evaluating this task. Reports Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.05169
  bibtex_key: bock2024weakannotation
  confidence: high
---

# weak-annotation-har-eval

> Weak-Annotation of HAR Datasets using Vision Foundation Models — Bock et al. (2024) (arXiv:2408.05169, 2024)

## What this evaluates

Evaluates inertial-based activity recognition models trained on weakly-supervised labels generated via vision foundation model clustering, benchmarked against fully-supervised and few-shot baselines.

## Datasets

- **WEAR** — total ?; splits: test (-1)
- **Wetlab** — total ?; splits: test (-1)
- **ActionSense** — total ?; splits: test (-1)

## Metrics

- `Acc` **(primary)** — range: percent
  - Fraction of correctly predicted activity labels out of total instances.
- `F1` — range: percent
  - Macro-averaged F1 score across all activity classes, including the NULL class.

## Input / output format

**Input**: 4-second IMU sensor data clips with 3-second overlap.

**Output**: Activity class label (including NULL-class).

## Scoring recipe

```python
def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    classes = set(golds)
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    f1 = sum(f1_scores) / len(f1_scores)
    return acc, f1
```

## Common pitfalls

- NULL-class has low intra-class similarity and is frequently misclassified as other activities.
- ActionSense dataset labels are vision-centric, making IMU-only classification inherently difficult.
- Distance thresholding removes 50–90% of training data, drastically reducing sample size.

## Evidence (verbatim from paper)

> Table 3 provides an overview across the eight evaluated training scenarios. Our proposed weakly-supervised training is not only capable of outperforming the few-shot training using only the annotated centroid clips, but for the case of applying 100 clusters is close to matching accuracy scores of a fully-supervised training across all three benchmark datasets, for both inertial-based architectures.

## Citation

```bibtex
@misc{bock2024weakannotation,
  title={Weak-Annotation of HAR Datasets using Vision Foundation Models},
  author={Bock et al. (2024)},
  year={2024},
  note={arXiv:2408.05169}
}
```

- arXiv: 2408.05169