Rospr Eval

---
name: rospr-eval
description: Evaluates the zero-shot generalization capability of instruction-tuned language models by retrieving and applying task-specific soft prompt embeddings at inference time to adapt to unseen tasks. Use when the user wants to benchmark on BIG-bench, SuperGLUE/HellaSwag/StoryCloze/WiC suite, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.03029
  bibtex_key: ye2022rospr
  confidence: medium
---

# rospr-eval

> Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Prompt — Ye et al. (2022) (arXiv:2210.03029, 2022)

## What this evaluates

Evaluates the zero-shot generalization capability of instruction-tuned language models by retrieving and applying task-specific soft prompt embeddings at inference time to adapt to unseen tasks.

## Datasets

- **BIG-bench** — total ?; splits: test (-1)
- **SuperGLUE/HellaSwag/StoryCloze/WiC suite** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answer choices or generated outputs matching the gold labels across all tasks.

## Input / output format

**Input**: Hard prompt template concatenated with the input instance, followed by a retrieved soft prompt embedding vector.

**Output**: Predicted answer choice or text generation corresponding to the hard prompt's expected output format.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize(pred) == normalize(gold):
        correct += 1
    total += 1
return (correct / total) * 100
```

## Common pitfalls

- Retrieval is performed on a randomly sampled subset of target task instances (Q) rather than the full test set, which may introduce variance.
- Soft prompts are trained on source tasks and retrieved via dense embedding similarity, which may not perfectly align with target task semantics or answer choice formats.

## Evidence (verbatim from paper)

> improving performance on 10 out of 11 tasks and boosting BIG-bench mean accuracy by 2.39% with only 0.007% additional parameters.

## Citation

```bibtex
@misc{ye2022rospr,
  title={Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Prompt},
  author={Ye et al. (2022)},
  year={2022},
  note={arXiv:2210.03029}
}
```

- arXiv: 2210.03029