Superni Performance Prediction Eval

---
name: superni-performance-prediction-eval
description: Evaluates the ability of a predictor model to estimate the performance of instruction-following language models on unseen tasks, using only the task instruction as input. It probes the fundamental challenge of third-party model transparency and controllability at the task level. Use when the user wants to benchmark on SuperNI, or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.12413
  bibtex_key: nadkarni2024instrperfpred
  confidence: high
---

# superni-performance-prediction-eval

> Third-Party Language Model Performance Prediction from Instruction — Nadkarni et al. (2024) (arXiv:2403.12413, 2024)

## What this evaluates

Evaluates the ability of a predictor model to estimate the performance of instruction-following language models on unseen tasks, using only the task instruction as input. It probes the fundamental challenge of third-party model transparency and controllability at the task level.

## Datasets

- **SuperNI** — total ?; splits: test (-1); repo https://github.com/rahuln/instr-perf-pred

## Metrics

- `RMSE` **(primary)** — range: other
  - Root Mean Squared Error between the predicted performance score and the true performance score across all test tasks. Computed as sqrt(mean((predicted - true)^2)).

## Input / output format

**Input**: Task instruction (string), optionally accompanied by two positive demonstrations (string) for few-shot prompting.

**Output**: A single floating-point number representing the predicted performance score (e.g., Exact Match or ROUGE-L percentage, or cross-entropy loss).

## Scoring recipe

```python
def compute_rmse(predictions, ground_truth):
    errors = [(p - g) ** 2 for p, g in zip(predictions, ground_truth)]
    return math.sqrt(sum(errors) / len(errors))

# Protocol: Train predictor on instruction-metric pairs, evaluate on 10 random splits of SuperNI test set, report mean RMSE ± std.
```

## Common pitfalls

- Confusing the target metric being predicted (Exact Match, ROUGE-L, or Loss) with the evaluation metric used to assess the predictor (RMSE).
- Assuming that increasing the size of the predictor model or adding diverse training tasks (e.g., BIG-bench) will significantly improve prediction accuracy, as domain shift and lack of learnable signal limit gains.
- Expecting the predictor to generalize across different instruction formats or model families without retraining, as performance predictability varies significantly by model and metric.

## Evidence (verbatim from paper)

> Table 1 shows our main results of predicting performance on SuperNI test set instructions, with mean and standard deviation (subscript) RMSE values across all train-test splits of the SuperNI test tasks. We explore using both Exact Match and ROUGE-L as the target metrics to be predicted for each task, with base and large RoBERTa models as the PP models as well as LLaMA-13B (upper bound) and the simple mean baseline (lower bound).

## Citation

```bibtex
@misc{nadkarni2024instrperfpred,
  title={Third-Party Language Model Performance Prediction from Instruction},
  author={Nadkarni et al. (2024)},
  year={2024},
  note={arXiv:2403.12413}
}
```

- arXiv: 2403.12413