Trglue Eval

---
name: trglue-eval
description: This benchmark evaluates Turkish natural language understanding across multiple task types, including grammaticality judgment, sentiment analysis, paraphrase detection, and semantic textual similarity. It probes a model's ability to handle agglutinative morphology, culturally adapted expressions, and nuanced semantic equivalence in Turkish. Use when the user wants to benchmark on TrCoLA, TrSST-2, TrMRPC, TrSTS-B, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.22100
  bibtex_key: altinok2025trglue
  confidence: high
---

# trglue-eval

> Introducing TrGLUE and SentiTurca: A Comprehensive Benchmark for Turkish General Language Understanding and Sentiment Analysis — Altinok (2025) (arXiv:2512.22100, 2025)

## What this evaluates

This benchmark evaluates Turkish natural language understanding across multiple task types, including grammaticality judgment, sentiment analysis, paraphrase detection, and semantic textual similarity. It probes a model's ability to handle agglutinative morphology, culturally adapted expressions, and nuanced semantic equivalence in Turkish.

## Datasets

- **TrCoLA** — total 9916; splits: train (-1), dev (-1), test (-1); HF `turkish-nlp-suite/TrCoLA`; repo https://github.com/turkish-nlp-suite/TrGLUE
- **TrSST-2** — total 78000; splits: train (67000), val (8900), test (8900); HF `turkish-nlp-suite/BuyukSinema`; repo https://github.com/turkish-nlp-suite/TrGLUE
- **TrMRPC** — total 5180; splits: test (-1); repo https://github.com/turkish-nlp-suite/TrGLUE
- **TrSTS-B** — total 3060; splits: dev (-1); repo https://github.com/turkish-nlp-suite/TrGLUE

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted labels out of the total number of instances. Standard for classification tasks in the benchmark.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall. Used alongside accuracy to address class imbalance in sentiment analysis tasks.
- `Spearman correlation` — range: [-1, 1]
  - Rank-based correlation coefficient used to evaluate the monotonic relationship between predicted and gold similarity scores on TrSTS-B.

## Input / output format

**Input**: Single sentence for TrCoLA and TrSST-2; sentence pair for TrMRPC, TrSTS-B, and TrQQP.

**Output**: Binary label (0 or 1) for TrCoLA, TrSST-2, TrMRPC, and TrQQP; continuous similarity score (1–5) for TrSTS-B.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- TrMRPC uses a relaxed paraphrase definition where sentences sharing the same main claim are labeled equivalent even if one contains extra non-contradictory details, diverging from strict English MRPC rules.
- TrSST-2 exhibits significant class imbalance in its original star ratings, requiring evaluation with both accuracy and F1-score rather than accuracy alone.
- TrSTS-B evaluation is reported exclusively on the development split because the test set was withheld due to the dataset's small size (3.06K pairs).

## Evidence (verbatim from paper)

> To address this class imbalance, we utilize the both binary accuracy and F1-score, whereas the original GLUE task employs binary accuracy.

## Citation

```bibtex
@misc{altinok2025trglue,
  title={Introducing TrGLUE and SentiTurca: A Comprehensive Benchmark for Turkish General Language Understanding and Sentiment Analysis},
  author={Altinok (2025)},
  year={2025},
  note={arXiv:2512.22100}
}
```

- arXiv: 2512.22100