Turkish Sseg Eval

---
name: turkish-sseg-eval
description: Evaluates the detection of sentence boundaries in Turkish text across diverse domains (scientific abstracts, news, social media). It tests robustness to formatting variations and punctuation absence. Use when the user wants to benchmark on trseg-41, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.01215
  bibtex_key: safaya2022mukayese
  confidence: high
---

# turkish-sseg-eval

> Mukayese: Turkish NLP Strikes Back — Safaya et al. (2022) (arXiv:2203.01215, 2022)

## What this evaluates

Evaluates the detection of sentence boundaries in Turkish text across diverse domains (scientific abstracts, news, social media). It tests robustness to formatting variations and punctuation absence.

## Datasets

- **trseg-41** — total 10700; splits: train (-1), dev (-1), test (-1); repo https://github.com/alisafaya/mukayese

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Segment-level F1 score computed over predicted vs gold sentence boundaries.
- `Precision` — range: [0, 1]
  - Segment-level precision for sentence boundary detection.
- `Recall` — range: [0, 1]
  - Segment-level recall for sentence boundary detection.

## Input / output format

**Input**: Raw text documents or social media posts.

**Output**: List of sentence boundary indices or segmented sentences.

## Scoring recipe

```python
def score_sseg(predictions, gold):
    pred_segments = model.predict_boundaries(text)
    gold_segments = gold_boundaries
    # Compute segment-level metrics
    p = len(set(pred_segments) & set(gold_segments)) / len(pred_segments) if pred_segments else 0
    r = len(set(pred_segments) & set(gold_segments)) / len(gold_segments) if gold_segments else 0
    f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
    return f1, p, r
```

## Common pitfalls

- Evaluation is done on the entire test set without removing ambiguous boundaries, unlike some prior work.
- Corrupted test sets (lowercased/punctuation removed) are used to stress-test robustness, which drastically lowers scores.
- Single annotator labels may introduce subjectivity not captured by automated metrics.

## Evidence (verbatim from paper)

> Our evaluation procedure is based on the metrics F1 score, Precision, Recall for each segment. Unlike Wicks and Post (2021), we evaluate our models on the entire test set, without removing sentences with ambiguous boundaries.

## Citation

```bibtex
@misc{safaya2022mukayese,
  title={Mukayese: Turkish NLP Strikes Back},
  author={Safaya et al. (2022)},
  year={2022},
  note={arXiv:2203.01215}
}
```

- arXiv: 2203.01215