Quoter Eval

---
name: quoter-eval
description: Evaluates a model's ability to recommend relevant quotes for a given writing context. It probes cross-lingual recommendation capabilities across English, Standard Chinese, and Classical Chinese by measuring how accurately and highly a model ranks the correct quote among a pool of candidates. Use when the user wants to benchmark on QuoteR, or asks about evaluating this task. Reports MRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2202.13145
  bibtex_key: qi2022quoter
  confidence: high
---

# quoter-eval

> QuoteR: A Benchmark of Quote Recommendation for Writing — Qi et al. (2022) (arXiv:2202.13145, 2022)

## What this evaluates

Evaluates a model's ability to recommend relevant quotes for a given writing context. It probes cross-lingual recommendation capabilities across English, Standard Chinese, and Classical Chinese by measuring how accurately and highly a model ranks the correct quote among a pool of candidates.

## Datasets

- **QuoteR** — total ?; splits: (unstated); repo https://github.com/thunlp/QuoteR

## Metrics

- `MRR` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank: the average of 1/rank(gold) across all query contexts. Higher is better.
- `NDCG@5` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at K=5. Computed as sum_{i=1}^5 (2^{r(i)}-1)/log2(i+1) with normalization constant Z_5=1, where r(i)=1 if the i-th ranked quote is gold, else 0. Paper specifically reports NDCG@5.
- `Recall@K` — range: [0, 1]
  - Proportion of query contexts whose gold quote appears in the top K ranked candidates. Evaluated at K=1, 10, and 100.

## Input / output format

**Input**: A query context (left context) and a candidate set of quotes.

**Output**: A ranked list of candidate quotes or relevance scores used to produce the ranking.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    mrr, ndcg5, recalls = 0.0, 0.0, {k: 0.0 for k in [1, 10, 100]}
    for pred_list, gold in zip(predictions, golds):
        rank = pred_list.index(gold) + 1
        mrr += 1.0 / rank
        if rank <= 5:
            ndcg5 += 1.0 / math.log2(rank + 1)
        for k in [1, 10, 100]:
            if rank <= k:
                recalls[k] += 1.0
    n = len(predictions)
    return {
        'MRR': mrr / n,
        'NDCG@5': ndcg5 / n,
        **{f'Recall@{k}': recalls[k] / n for k in [1, 10, 100]}
    }
```

## Common pitfalls

- The paper specifically reports NDCG@5, not the more common NDCG@10 or NDCG@20.
- Rank-based metrics (Median Rank, Mean Rank, Rank Variance) are minimized, whereas MRR, NDCG, and Recall are maximized; confusing the optimization direction leads to incorrect model selection.
- Models must be evaluated separately on the English, Standard Chinese, and Classical Chinese splits, as Chinese BERT is pre-trained on standard Chinese and performs poorly on classical Chinese without adaptation.

## Evidence (verbatim from paper)

> Following previous work (Ahn et al., 2016; Tan et al., 2018), we use three evaluation metrics: (1) Mean reciprocal rank (MRR), the average reciprocal values of the ranks of the gold quotes; (2) Normalized discounted cumulative gain (NDCG@K) (Järvelin and Kekäläinen, 2002), a widely used measure of ranking quality and is computed by ... We report the average of NDCG@5 scores of all the evaluated query contexts. (3) Recall@K, the proportion of query contexts whose gold quotes are ranked in respective top K candidate quotes, K={1,10,100}.

## Citation

```bibtex
@misc{qi2022quoter,
  title={QuoteR: A Benchmark of Quote Recommendation for Writing},
  author={Qi et al. (2022)},
  year={2022},
  note={arXiv:2202.13145}
}
```

- arXiv: 2202.13145