Q Measure

---
name: q-measure
description: Evaluates ranked retrieval lists using graded relevance assessments, balancing precision and cumulative gain while penalizing lower-ranked relevant documents. Use when the user has predictions and gold and needs to compute Q-measure.
metadata:
  skill_kind: metric
  source_arxiv: 1903.11272
  bibtex_key: sakai2019graded
  confidence: high
---

# q-measure

> Graded Relevance Assessments and Graded Relevance Measures of NTCIR: A Survey of the First Twenty Years — Sakai et al. (2019) (arXiv:1903.11272, 2019)

## What this evaluates

Evaluates ranked retrieval lists using graded relevance assessments, balancing precision and cumulative gain while penalizing lower-ranked relevant documents.

## Datasets

- (no dataset; pure metric skill)

## Metrics

- `Q-measure` **(primary)** — range: [0, 1]
  - Q = (1/R) * sum_{r} I(r) * BR(r), where BR(r) = (C(r) + beta * cg(r)) / (r + beta * cg*(r)). C(r) is binary precision count, cg(r) is cumulative graded gain, cg*(r) is cumulative gain of ideal list, R is total relevant docs, beta is patience (usually 1). Reduces to AP when beta=0.

## Input / output format

**Input**: A ranked list of retrieved documents/items for a query, along with graded relevance labels (e.g., 0=nonrelevant, 1=partially relevant, 2=highly relevant) for each item.

**Output**: A single scalar score in [0, 1] representing the retrieval quality.

## Scoring recipe

```python
def q_measure(retrieved_docs, relevance_labels, beta=1.0):
    R = sum(1 for x in relevance_labels if x > 0)
    if R == 0: return 0.0
    gains = [gain_value[x] for x in relevance_labels]
    ideal_gains = sorted(gains, reverse=True)
    C = 0; cg = 0; cg_star = 0
    Q = 0.0
    for r, rel in enumerate(relevance_labels, 1):
        I_r = 1 if rel > 0 else 0
        C += I_r
        cg += gains[r-1]
        cg_star += ideal_gains[r-1]
        BR = (C + beta * cg) / (r + beta * cg_star)
        Q += I_r * BR
    return Q / R
```

## Common pitfalls

- Equivalence classes for answer strings (e.g., in QA tasks) are handled by counting only one relevant string per class, unlike standard document retrieval.
- The patience parameter beta is usually set to 1 but controls the trade-off between precision and nCG; setting beta=0 reduces Q to AP.
- Q and nDCG behave similarly but use different mechanisms to penalize low-rank relevant documents (denominator rank vs. logarithmic discount).

## Evidence (verbatim from paper)

> The Q-measure is defined as Q = (1/R) sum_{r} I(r) BR(r), where BR(r) is the blended ratio given by BR(r) = (C(r) + beta cg(r)) / (r + beta cg*(r)). Here, beta is the patience parameter which is usually set to one; its significance is discussed in Sakai (2014). Note that C(r)/r represents binary Precision at rank r; hence, both precision and nCG are embedded in Eq. 2.

## Citation

```bibtex
@misc{sakai2019graded,
  title={Graded Relevance Assessments and Graded Relevance Measures of NTCIR: A Survey of the First Twenty Years},
  author={Sakai et al. (2019)},
  year={2019},
  note={arXiv:1903.11272}
}
```

- arXiv: 1903.11272