Filler Word Detection Eval

---
name: filler-word-detection-eval
description: Evaluates a model's ability to detect and classify filler words (e.g., 'uh', 'um') in naturalistic speech recordings. It probes temporal localization accuracy and fine-grained acoustic classification under varying evaluation granularities. Use when the user wants to benchmark on PodcastFillers, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.15135
  bibtex_key: zhu2022fillerword
  confidence: high
---

# filler-word-detection-eval

> Filler Word Detection and Classification: A Dataset and Benchmark — Zhu et al. (2022) (arXiv:2203.15135, 2022)

## What this evaluates

Evaluates a model's ability to detect and classify filler words (e.g., 'uh', 'um') in naturalistic speech recordings. It probes temporal localization accuracy and fine-grained acoustic classification under varying evaluation granularities.

## Datasets

- **PodcastFillers** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `F1` **(primary)** — range: percent
  - Harmonic mean of Precision and Recall: F1 = 2 * (P * R) / (P + R). Reported at both segment-level and event-level granularities.
- `Precision` — range: percent
  - Ratio of correctly detected/classified filler instances to all instances predicted as filler.
- `Recall` — range: percent
  - Ratio of correctly detected/classified filler instances to all ground truth filler instances.

## Input / output format

**Input**: Raw audio recordings processed into acoustic features (wav2vec or log-mel embeddings) and optionally ASR transcriptions.

**Output**: Binary or multi-class classification labels ('uh', 'um', or other) with temporal boundaries at either the segment level or event level.

## Scoring recipe

```python
def compute_f1(preds, gold, granularity='event'):
    tp = sum(1 for p in preds if any(match(p, g, granularity) for g in gold))
    fp = len(preds) - tp
    fn = len(gold) - tp
    p = tp / (tp + fp) if (tp + fp) > 0 else 0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * p * r / (p + r) if (p + r) > 0 else 0
```

## Common pitfalls

- Evaluation is reported at two different granularities (segment vs. event); results are not directly comparable across granularities.
- AVC-FillerNet relies on ASR output to generate candidates, making its performance dependent on ASR quality, whereas VC-FillerNet is transcription-free.
- VAD threshold significantly impacts recall; a low threshold (0.1) is required to maximize recall without hurting precision.

## Evidence (verbatim from paper)

> Table 2: Segment- and event-based F1 measure (%) results for separately detecting 'uh' and 'um' with our proposed systems.

## Citation

```bibtex
@misc{zhu2022fillerword,
  title={Filler Word Detection and Classification: A Dataset and Benchmark},
  author={Zhu et al. (2022)},
  year={2022},
  note={arXiv:2203.15135}
}
```

- arXiv: 2203.15135