Leaderboard Zero Shot Rte Eval

---
name: leaderboard-zero-shot-rte-eval
description: Evaluates whether pre-trained Recognizing Textual Entailment (RTE) models can generalize to unseen task-dataset-metric (TDM) extraction pairs in a zero-shot setting. It probes whether models learn genuine semantic entailment or merely memorize training distribution patterns. Use when the user wants to benchmark on LEADERBOARDS, or asks about evaluating this task. Reports macro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.16835
  bibtex_key: kabongo2023leaderboard
  confidence: high
---

# leaderboard-zero-shot-rte-eval

> Zero-shot Entailment of Leaderboards for Empirical AI Research — Kabongo et al. (2023) (arXiv:2303.16835, 2023)

## What this evaluates

Evaluates whether pre-trained Recognizing Textual Entailment (RTE) models can generalize to unseen task-dataset-metric (TDM) extraction pairs in a zero-shot setting. It probes whether models learn genuine semantic entailment or merely memorize training distribution patterns.

## Datasets

- **LEADERBOARDS** — total ?; splits: test (-1)

## Metrics

- `macro F1` **(primary)** — range: percent
  - Unweighted mean of the F1 scores computed for each class independently.
- `micro F1` — range: percent
  - Global F1 score calculated by aggregating true positives, false positives, and false negatives across all classes before computing precision and recall.

## Input / output format

**Input**: A document representation (DocTAET T) paired with a Task-Dataset-Metric (TDM) hypothesis for sequence-pair classification.

**Output**: Classification label indicating whether the document entails the TDM hypothesis.

## Scoring recipe

```python
def compute_f1(y_true, y_pred, average='macro'):
    # Calculate per-class or global TP, FP, FN
    # precision = TP / (TP + FP)
    # recall = TP / (TP + FN)
    # f1 = 2 * (precision * recall) / (precision + recall)
    # Return macro average (mean of per-class F1) or micro average (global aggregation)
    return f1_score(y_true, y_pred, average=average) * 100
```

## Common pitfalls

- Models exhibit very high recall but extremely low precision, indicating they classify most hypotheses as true rather than identifying the correct entailment.
- The task is framed as RTE but functions as multi-class classification; models may memorize training patterns instead of learning true semantic entailment.
- Longer document representations significantly degrade performance, suggesting shorter (1-2 sentence) summaries are necessary for effective entailment.

## Evidence (verbatim from paper)

> ORKG-TDM $_{Bert}$ in a non-zero-shot setting reported 90.8% macro F1 and 91.8% micro F1 [14]. The same model in a zero-shot setting reports performances significantly less than chance at 26.7% macro F1 and 21% micro F1. Likewise, the other SOTA model ORKG-TDM $_{XLNet}$ in a non-zero-shot setting reported 91.2% macro F1 and 92.4% micro F1. This model in the zero-shot setting reported 22.3% macro F1 and 17.2% micro F1.

## Citation

```bibtex
@misc{kabongo2023leaderboard,
  title={Zero-shot Entailment of Leaderboards for Empirical AI Research},
  author={Kabongo et al. (2023)},
  year={2023},
  note={arXiv:2303.16835}
}
```

- arXiv: 2303.16835