Gptscore Eval

---
name: gptscore-eval
description: Evaluates the correlation between automated scoring functions (GPTScore variants) and human judgments across multiple text generation tasks. It probes the ability of instruction-based LLMs to serve as training-free, customizable evaluators that align with human preference. Use when the user wants to benchmark on SummEval, RealSumm, NEWSROOM, QXSUM, MQM-2020, BAGEL, SFRES, FED, or asks about evaluating this task. Reports Spearman correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.04166
  bibtex_key: fu2023gptscore
  confidence: high
---

# gptscore-eval

> GPTScore: Evaluate as You Desire — Jinlan Fu et al. (2023) (arXiv:2302.04166, 2023)

## What this evaluates

Evaluates the correlation between automated scoring functions (GPTScore variants) and human judgments across multiple text generation tasks. It probes the ability of instruction-based LLMs to serve as training-free, customizable evaluators that align with human preference.

## Datasets

- **SummEval** — total ?; splits: test (-1)
- **RealSumm** — total ?; splits: test (-1)
- **NEWSROOM** — total ?; splits: test (-1)
- **QXSUM** — total ?; splits: test (-1)
- **MQM-2020** — total ?; splits: test (-1)
- **BAGEL** — total ?; splits: test (-1)
- **SFRES** — total ?; splits: test (-1)
- **FED** — total ?; splits: test (-1)

## Metrics

- `Spearman correlation` **(primary)** — range: [-1, 1]
  - Standard rank correlation coefficient measuring the monotonic relationship between the model's predicted scores (conditional generation probabilities) and human judgment scores. Computed as ρ = 1 - (6Σd_i²)/(n(n²-1)) for tied ranks, or via standard library rankdata.

## Input / output format

**Input**: Source text, reference text, and generated hypothesis (or system output).

**Output**: A continuous score representing the conditional generation probability of the reference given the source and hypothesis, or vice versa, depending on the prompt configuration.

## Scoring recipe

```python
def compute_spearman(pred_scores, human_scores):
    # pred_scores: list of GPTScore probabilities for each instance
    # human_scores: list of human judgment scores for each instance
    rank_pred = rankdata(pred_scores)
    rank_human = rankdata(human_scores)
    n = len(rank_pred)
    d_sq = sum((r1 - r2)**2 for r1, r2 in zip(rank_pred, rank_human))
    rho = 1 - (6 * d_sq) / (n * (n**2 - 1))
    return rho
```

## Common pitfalls

- GPTScore outputs conditional probabilities, not direct human-like scores; evaluation requires computing correlation against human judgments rather than direct accuracy.
- The paper reports correlation coefficients, so lower scores indicate worse alignment with human preference, not worse generation quality.
- Instruction and demonstration settings (VAL, IST, IDM) significantly impact results; vanilla prompts often underperform and should not be used for final reporting.

## Evidence (verbatim from paper)

> Fig. 3 shows the evaluation results of five GPT3 variant models on four text summarization datasets, where QXSUM uses the Pearson correlation and other datasets use the Spearman correlation metric.

## Citation

```bibtex
@misc{fu2023gptscore,
  title={GPTScore: Evaluate as You Desire},
  author={Jinlan Fu et al. (2023)},
  year={2023},
  note={arXiv:2302.04166}
}
```

- arXiv: 2302.04166