Ytseg Segmentation Eval

---
name: ytseg-segmentation-eval
description: Evaluates a model's ability to detect topic boundaries in unstructured spoken transcriptions (text segmentation) and generate coherent chapter titles (smart chaptering). It probes hierarchical structuring, real-time/online processing constraints, and cross-domain generalization to meeting transcripts. Use when the user wants to benchmark on WIKI-727K, YTSEG, QMSUM, YTSEG[TITLES], or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.17633
  bibtex_key: retkowski2024smartchaptering
  confidence: high
---

# ytseg-segmentation-eval

> From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions — Retkowski et al. (2024) (arXiv:2402.17633, 2024)

## What this evaluates

Evaluates a model's ability to detect topic boundaries in unstructured spoken transcriptions (text segmentation) and generate coherent chapter titles (smart chaptering). It probes hierarchical structuring, real-time/online processing constraints, and cross-domain generalization to meeting transcripts.

## Datasets

- **WIKI-727K** — total ?; splits: test (-1)
- **YTSEG** — total ?; splits: test (-1)
- **QMSUM** — total 232; splits: test (-1)
- **YTSEG[TITLES]** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for binary boundary detection: 2 * (P * R) / (P + R).
- `Pk` — range: [0, 1]
  - Boundary error rate computed over sliding windows; lower values indicate better boundary alignment.
- `Boundary Similarity (B1)` — range: [0, 1]
  - Measures overlap between predicted and gold boundary sets, penalizing mismatches based on distance.
- `ROUGE-1/2/L` — range: [0, 1]
  - Lexical overlap metrics measuring unigram, bigram, and longest common subsequence overlap between generated and gold titles.
- `BARTScore` — range: other
  - Semantic similarity score computed using a fine-tuned BART model; higher values indicate better semantic equivalence.

## Input / output format

**Input**: Sequence of sentences/text segments from video transcriptions or meeting recordings, optionally augmented with previous chapter titles or a fixed future context window (c) for online evaluation.

**Output**: Binary boundary predictions (segment start/end indices) for segmentation; generated chapter titles for title generation.

## Scoring recipe

```python
def score_segmentation(pred_boundaries, gold_boundaries):
    P = len(pred & gold) / len(pred) if pred else 0
    R = len(pred & gold) / len(gold) if gold else 0
    F1 = 2 * P * R / (P + R) if (P + R) else 0
    Pk = compute_pk_error(pred, gold, window=10)
    B1 = compute_boundary_similarity(pred, gold)
    return {'P': P, 'R': R, 'F1': F1, 'Pk': Pk, 'B1': B1}

def score_titles(gen_titles, gold_titles):
    R1 = rouge_score(gen_titles, gold_titles, rouge_types=['rouge1'])
    BS = bartscore_metric(gen_titles, gold_titles)
    return {'R1': R1, 'BS': BS}
```

## Common pitfalls

- QMSUM's small size (232 meetings) makes performance metrics non-robust and cross-dataset transfer results inconclusive.
- Online/real-time evaluation requires careful balancing of future context size (c) and input span (s) to manage latency vs. performance trade-offs.
- Title generation without prior context leads to repetitive functional titles (e.g., 'Intro') and poor stylistic coherence across chapters.

## Evidence (verbatim from paper)

> We evaluate our segmentation models using a combination of standard binary classification metrics, such as precision, recall, and F1 score, as well as metrics specifically tailored for text segmentation tasks, including Pk as introduced in the work of Beeferman et al. (1999) and Boundary Similarity, as discussed in Fournier (2013). The generation of section titles can be considered an extreme form of summarization. As such, we evaluate our models using established metrics in summarization: ROUGE (Lin, 2004), which measures the lexical overlap, and BARTScore (Yuan et al., 2021), an increasingly used metric for semantic equivalence.

## Citation

```bibtex
@misc{retkowski2024smartchaptering,
  title={From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions},
  author={Retkowski et al. (2024)},
  year={2024},
  note={arXiv:2402.17633}
}
```

- arXiv: 2402.17633