Tape Eval

---
name: tape-eval
description: Evaluates few-shot and zero-shot Russian language understanding across six tasks probing logical reasoning, multi-hop inference, commonsense knowledge, and ethical judgment. It also measures model robustness against linguistic adversarial perturbations like typos, deletions, and modality changes. Use when the user wants to benchmark on TAPE, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.12813
  bibtex_key: taktasheva2022tape
  confidence: high
---

# tape-eval

> TAPE: Assessing Few-shot Russian Language Understanding — Taktasheva et al. (2022) (arXiv:2210.12813, 2022)

## What this evaluates

Evaluates few-shot and zero-shot Russian language understanding across six tasks probing logical reasoning, multi-hop inference, commonsense knowledge, and ethical judgment. It also measures model robustness against linguistic adversarial perturbations like typos, deletions, and modality changes.

## Datasets

- **TAPE** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels or multiple-choice answers. Calculated as the number of matches between predictions and gold labels divided by total instances.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall, reported as weighted average for imbalanced classification tasks.
- `ASR` — range: percent
  - Attack Success Rate, measuring the percentage drop in model performance after applying adversarial perturbations compared to the clean baseline.

## Input / output format

**Input**: Russian text passages or questions, optionally preceded by k-shot demonstrations (k ∈ {1, 4, 8}) formatted as prompt examples.

**Output**: Predicted class labels for classification/multiple-choice tasks; free-form generated text for QA tasks.

## Scoring recipe

```python
# Classification/Multiple-Choice
acc = sum(pred == gold for pred, gold in zip(predictions, golds)) / len(golds)
f1 = f1_score(golds, predictions, average='weighted')
# QA Generation
exact_match = sum(gen.strip() == gold.strip() for gen, gold in zip(generated_preds, golds)) / len(golds)
# Robustness (ASR)
asr = ((p_clean - p_perturbed) / p_clean) * 100
```

## Common pitfalls

- Few-shot demonstrations can cause performance degradation or constant predictions on imbalanced tasks (e.g., models predict the negative label ~97% of the time in zero-shot).
- Standard exact-match QA metrics fail to capture semantically correct but non-literal generated answers, requiring manual semantic analysis.
- Model performance exhibits strong length bias, dropping significantly on longer inputs due to context window limitations.

## Evidence (verbatim from paper)

> Table 4 shows the ASR scores for each perturbation and k-shot setting averaged over the RuWorldTree and RuOpenBookQA tasks, where the model performance exceeds the random baseline. We observe that the models are more susceptible to simple spelling-based perturbations (BUTTERFINGERS), token deletion (EDA_DELETE) and modality changes (EMOJIFY).

## Citation

```bibtex
@misc{taktasheva2022tape,
  title={TAPE: Assessing Few-shot Russian Language Understanding},
  author={Taktasheva et al. (2022)},
  year={2022},
  note={arXiv:2210.12813}
}
```

- arXiv: 2210.12813