Btzsc Eval

---
name: btzsc-eval
description: Evaluates zero-shot text classification capabilities across diverse datasets using four model families: NLI cross-encoders, embedding models, rerankers, and instruction-tuned LLMs. It probes how well models can assign text to predefined categories without task-specific fine-tuning by verbalizing labels as descriptions. Use when the user wants to benchmark on BTZSC Benchmark, or asks about evaluating this task. Reports macro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.11991
  bibtex_key: aarab2026btzsc
  confidence: high
---

# btzsc-eval

> BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs — Aarab (2026) (arXiv:2603.11991, 2026)

## What this evaluates

Evaluates zero-shot text classification capabilities across diverse datasets using four model families: NLI cross-encoders, embedding models, rerankers, and instruction-tuned LLMs. It probes how well models can assign text to predefined categories without task-specific fine-tuning by verbalizing labels as descriptions.

## Datasets

- **BTZSC Benchmark** — total ?; splits: test (-1); repo https://github.com/IliasAarab/btzsc

## Metrics

- `macro F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score computed across all classes and datasets, averaging per-class F1 scores equally regardless of class frequency.

## Input / output format

**Input**: Input text paired with a set of verbalized label descriptions (one per class).

**Output**: Predicted class label (for cross-encoders, embeddings, rerankers) or selected multiple-choice option (for LLMs).

## Scoring recipe

```python
def compute_macro_f1(predictions, gold_labels):
    f1_scores = []
    for label in set(gold_labels):
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == label and g == label)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == label and g != label)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != label and g == label)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Label verbalization is mandatory; using raw class names instead of context-rich descriptions will break zero-shot prompting.
- Different model families require distinct inference pipelines (logits for cross-encoders, cosine similarity for embeddings, relevance scores for rerankers, next-token probabilities for LLMs).
- Performance is aggregated across 22 diverse datasets, so reporting a single aggregate number without dataset-level breakdowns obscures domain-specific variations.

## Evidence (verbatim from paper)

> To facilitate zero-shot classification, each class label is verbalized as a short, semantically clear, and context-rich description. ... Results show rerankers (e.g., Qwen3-Reranker-8B) achieve state-of-the-art macro F1 of 0.72, embedding models (e.g., GTE-large-en-v1.5) offer optimal accuracy-latency trade-offs, and LLMs (4–12B params) perform competitively on topic classification but lag behind rerankers.

## Citation

```bibtex
@misc{aarab2026btzsc,
  title={BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs},
  author={Aarab (2026)},
  year={2026},
  note={arXiv:2603.11991}
}
```

- arXiv: 2603.11991