Judgment List Author

---
name: judgment-list-author
description: "Bootstraps human-relevance judgment lists (query sets, grading scales, rater guidelines, inter-rater agreement, Quepid tooling, TREC-style pooling, and refresh cadence) that serve as ground truth for search-relevance test suites. Use when a team needs to create or refresh the judgment corpus before running NDCG / MRR / Recall@k evaluations."
metadata:
  keywords: "judgment-list, qrels, relevance-ground-truth, quepid, trec, inter-rater-agreement, ndcg"
  version: "1.2.5"
---

# judgment-list-author

Related skills (`elasticsearch-relevance-tests` - covering Elasticsearch,
OpenSearch, and Solr - and `vector-search-recall-tests`) all require a
judgment list - a set of `(query, document_id, grade)` triples that define what
"relevant" means for your product. No automated metric creates that
corpus. This skill does.

Per [TREC's pooling methodology](https://trec.nist.gov/overview.html), "NIST
pools the individual results, judges the retrieved documents for correctness,
and evaluates the results" - the
judgment list is the non-automated step that all automated metrics depend on.

## When to use

- Before running `_rank_eval` or `recall@k` for the first time: no judgment
  list means no metrics.
- After a major schema or corpus change: existing judgments cover documents
  that may no longer exist or new documents that are entirely unjudged.
- When a relevance-regression check reports > 30% unrated docs across
  queries: the judgment pool is stale.
- When adding a new query segment (new product category, new language) with
  no coverage in the existing set.

## Step 1 - Select the query set (head / torso / tail sampling)

Not all queries deserve equal judgment effort. Sample across three tiers:

| Tier | Definition | Suggested count | Priority |
|---|---|---|---|
| Head | Top ~100-500 by traffic volume (covers ~80% of impressions) | 50-100 queries | Highest |
| Torso | Queries ranked ~500-5000 (navigational, faceted) | 100-200 queries | Medium |
| Tail | Low-frequency, long-tail queries | 50-100 queries | Lower |

Pull the sample from query logs, not from intuition. Use a 90-day window to
avoid seasonal skew. Keep the raw log row for each sampled query - you will
need it to assign traffic weight when computing weighted NDCG.

A starting corpus of roughly 200-400 queries across tiers gives sufficient
coverage for NDCG-based gates. Per
[Quepid docs](https://quepid-docs.dev.o19s.com/2/quepid), "100 judgments
(10 pages of 10 search results) is a good starting point and likely
sufficient for most cases".

## Step 2 - Choose the grading scale

Two options are widely used:

### Binary (TREC standard)

Per [TREC qrels format](https://trec.nist.gov/data/qrels_eng/), the classic
qrels file uses two values:
`0` (not relevant) and `1` (relevant). Simple to collect; sufficient for
Precision@k and Recall@k. Use binary when raters have low domain expertise
or when the query intent is unambiguous (navigational queries).

Qrels file format (4 columns, per
[TREC qrels format](https://trec.nist.gov/data/qrels_eng/)):

```text
<topic_id> <iteration> <doc_id> <relevance>
1 0 doc-abc-123 0
1 0 doc-xyz-456 1
```

The `iteration` column is "almost always zero and not used" per TREC.
Unjudged documents are assumed irrelevant in evaluation.

### Graded 0-3 (Quepid default)

Per [Quepid judgment rating best practices](https://github.com/o19s/quepid/wiki/Judgement-Rating-Best-Practices),
the 0-3 scale maps to these grades and customer-feeling cues (the source
frames them for e-commerce; restate them in your own domain's terms):

| Grade | Label | Rater cue |
|---|---|---|
| 3 | Perfect | "This is what I am looking for! I'm going to buy one." |
| 2 | Good | "I like these! Now I can look more and decide which one to buy." |
| 1 | Fair | "These results aren't what I'm looking for." |
| 0 | Poor | "These results are terrible! Maybe I'll look somewhere else." |

Use graded when you need NDCG or DCG (metrics that reward highly relevant
results at higher ranks). Required by `elasticsearch-relevance-tests` Step 1
which uses a 4-point (0-3) scale.

## Step 3 - Write grading guidelines for raters

Grading guidelines prevent scale drift between raters and across time. A
minimal guidelines document covers:

1. **Task definition:** state the user's information need, not just the
   query string. "query: running shoes" could be navigational (find Nike
   page) or informational (compare models).
2. **Grade anchors with examples:** for each grade level, list 2-3 concrete
   examples from your domain. Raters calibrate to examples faster than
   to prose definitions.
3. **Edge cases by domain:** e-commerce (out-of-stock items, variant SKUs),
   docs search (older versions, deprecated APIs), support search
   (resolved vs open tickets).
4. **What to do with unrateable documents:** Quepid supports an `unrateable`
   flag per [Quepid API](https://github.com/o19s/quepid/blob/main/app/controllers/api/v1/judgements_controller.rb)
   (permitted judgement params: `rating`, `unrateable`, `judge_later`,
   `query_doc_pair_id`, `user_id`, `explanation`). Use it for documents where
   the rater cannot determine relevance (e.g. page behind a login wall).

Store the guidelines in version control alongside the judgment file. When
guidelines change, treat it as a new judgment round - old and new grades
are not comparable.

## Step 4 - Set up Quepid for rater workflow

[Quepid](https://github.com/o19s/quepid) is the standard open-source tool for
collaborative judgment authoring. It runs as a self-hosted Rails app and
connects to Elasticsearch, OpenSearch, Solr, Algolia, and other backends.

Setup flow:

1. Deploy Quepid via Docker: `docker-compose up` per the Quepid README.
2. Create a **Case** - one Case = one judgment corpus for one search
   endpoint + query set.
3. Import queries (CSV or manual entry). Each query gets a displayed
   information-need description shown to the rater.
4. Assign raters. Quepid tracks `user_id` per judgment per
   [Quepid API](https://github.com/o19s/quepid/blob/main/app/controllers/api/v1/judgements_controller.rb),
   enabling per-rater analysis.
5. Raters work through the **Human Rating Interface**: query + information
   need + rendered document + grade selector.
6. Export when done. Per
   [Quepid API](https://github.com/o19s/quepid/blob/main/app/controllers/api/v1/judgements_controller.rb),
   the export is a CSV whose header row starts `query, docid` and then
   appends one column per judge name, with filename
   `book_{id}_judgements.csv`. JSON and Learning-to-Rank formats are also
   supported per [Quepid docs](https://quepid-docs.dev.o19s.com/2/quepid).

For teams without Quepid, a spreadsheet works for small sets (under 500
judgments): columns `query_id, query_text, doc_id, grade, rater_id, notes`.
Convert to qrels format before feeding `_rank_eval`.

## Step 5 - Measure inter-rater agreement (Cohen's kappa)

Have at least two independent raters judge the same 10-20% overlap set.
Compute Cohen's kappa to verify the scale is being applied consistently.

Formula per [Cohen's kappa, Wikipedia](https://en.wikipedia.org/wiki/Cohen%27s_kappa):

```
kappa = (p_o - p_e) / (1 - p_e)
```

where `p_o` is observed agreement and `p_e` is expected chance agreement.

Interpretation thresholds (Landis and Koch, 1977, as cited in
[Cohen's kappa, Wikipedia](https://en.wikipedia.org/wiki/Cohen%27s_kappa)):

| Kappa | Agreement |
|---|---|
| < 0.20 | Slight - raters are guessing; revise guidelines |
| 0.21-0.40 | Fair - guidelines unclear; calibrate with examples |
| 0.41-0.60 | Moderate - acceptable for exploratory work |
| 0.61-0.80 | Substantial - good for production gates |
| 0.81-1.00 | Almost perfect - target for high-stakes domains |

For binary judgments, kappa < 0.60 means your guidelines are ambiguous.
Rewrite the edge-case section, run a calibration session with raters, and
re-judge the overlap set before proceeding.

```python
from sklearn.metrics import cohen_kappa_score

rater_a = [3, 2, 1, 0, 3, 2, 2, 1, 0, 3]
rater_b = [3, 2, 0, 0, 3, 1, 2, 1, 0, 2]

kappa = cohen_kappa_score(rater_a, rater_b)
print(f"Cohen's kappa: {kappa:.3f}")
# 0.61-0.80 = substantial agreement; proceed to full rating round
```

When multiple raters cover non-overlapping document sets (common for large
judgment pools), use Krippendorff's alpha instead - it handles missing data
across raters.

## Step 6 - Pool from multiple retrieval systems

When you have results from more than one system (e.g. current BM25 +
candidate neural re-ranker), use TREC-style depth-k pooling to maximize
judgment coverage.

Per [TREC pooling, Wikipedia](https://en.wikipedia.org/wiki/Text_Retrieval_Conference),
in pooling "the top-ranked n documents from each contributing run are
aggregated, and the resulting document set is judged completely."

Practical pooling:

```python
def pool_results(system_results: dict[str, list[str]], depth: int = 10) -> set[str]:
    """
    system_results: { system_name: [doc_id, ...] } top-depth per query
    Returns the union of all doc IDs in the pool.
    """
    pool = set()
    for docs in system_results.values():
        pool.update(docs[:depth])
    return pool
```

Pool depth = 10 is standard for small collections (< 100k docs). Increase
to 20-50 when you have > 3 candidate systems to avoid missing relevant
documents that only appear in one system's lower ranks.

Documents outside the pool are unjudged. Per
[TREC qrels format](https://trec.nist.gov/data/qrels_eng/), documents "not
occurring in the qrels file were not judged by the human assessor and are
assumed to be irrelevant in the evaluations used in TREC." This is
conservative but consistent across systems.

## Step 7 - Establish refresh cadence

Judgment lists go stale when the document corpus changes substantially.
Define explicit refresh triggers:

| Trigger | Action |
|---|---|
| Index schema change (new field, new analyzer) | Full re-pool + partial re-judge (re-rate 20% overlap) |
| Embedding model upgrade | Full re-pool for all affected query tiers |
| Corpus grows > 20% | Re-pool; re-judge new documents only |
| A relevance-regression check flags > 30% unrated | Partial re-judge: new docs in the unrated set |
| New product category / language added | Add new query stratum; judge from scratch for that stratum |

As a minimum, run a lightweight staleness check monthly: for each query,
count the `unrated_docs` fraction in `_rank_eval` results. If the average
exceeds 20%, schedule a re-judging session.

## Output format

The judgment list consumed by `elasticsearch-relevance-tests` (for both
Elasticsearch and OpenSearch rank eval) is a JSON array:

```json
[
  {
    "query_id": "q001",
    "query": "running shoes",
    "ratings": [
      { "doc_id": "doc-abc", "rating": 3 },
      { "doc_id": "doc-xyz", "rating": 1 },
      { "doc_id": "doc-mno", "rating": 0 }
    ]
  }
]
```

Or equivalently as a TREC qrels file (4 columns) for binary cases, which
tools like `trec_eval` and the Elasticsearch `_rank_eval` API both accept
after mapping `doc_id` to the index's `_id` field.

## Anti-patterns

| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Judge only the current system's top-10 | Biases the corpus toward the incumbent; new systems retrieve different docs | Pool from all candidate systems (Step 6) |
| One rater, no kappa check | Silent scale drift; metrics become meaningless over time | Require 10-20% overlap + kappa >= 0.60 (Step 5) |
| Reuse judgments after embedding model upgrade | Vector space changed; doc rankings shift entirely | Re-pool and re-judge after any embedding change |
| Judge head queries only | Tail queries drive long-tail revenue; regressions go undetected | Sample across head/torso/tail (Step 1) |
| Treat unjudged docs as relevant | Inflates recall metrics artificially | Default unjudged to irrelevant per TREC convention |
| No version control on guidelines | Raters from two periods use different scales; grades are incompatible | Store guidelines in git; treat guideline changes as a new round |

## Limitations

- TREC pooling assumes multiple competing systems. Single-system shops
  have shallower pools and higher unjudged-doc rates.
- Graded 0-3 scale requires domain-expert raters. Binary scale is
  appropriate when domain expertise is scarce.
- Cohen's kappa understates agreement on rare categories (e.g. grade 3
  "perfect" is rare for tail queries). Per
  [Cohen's kappa, Wikipedia](https://en.wikipedia.org/wiki/Cohen%27s_kappa),
  kappa "tends to underestimate the agreement on the rare category" -
  supplement with per-grade agreement counts when the kappa is borderline.
- Judgment effort scales with query count and pool depth. A 400-query set
  at depth 20 = 8,000 documents to judge; plan rater time accordingly.

## References

- [TREC qrels format] - 4-column qrels file format, binary relevance scale,
  pooling convention: https://trec.nist.gov/data/qrels_eng/
- [TREC pooling, Wikipedia] - depth-k pooling methodology:
  https://en.wikipedia.org/wiki/Text_Retrieval_Conference
- [Quepid] - open-source judgment authoring tool (Rails, supports ES /
  OpenSearch / Solr / Algolia): https://github.com/o19s/quepid
- [Quepid judgment rating best practices] - 0-3 grade scale (Poor / Fair /
  Good / Perfect) with per-grade rater cues:
  https://github.com/o19s/quepid/wiki/Judgement-Rating-Best-Practices
- [Quepid docs] - rater workflow, 100-judgment baseline, export formats
  (CSV / JSON / LTR): https://quepid-docs.dev.o19s.com/2/quepid
- [Quepid API] - judgment fields (`rating`, `unrateable`, `judge_later`,
  `explanation`), CSV export structure `book_{id}_judgements.csv`:
  https://github.com/o19s/quepid/blob/main/app/controllers/api/v1/judgements_controller.rb
- [Cohen's kappa, Wikipedia] - kappa formula, Landis and Koch (1977)
  thresholds: https://en.wikipedia.org/wiki/Cohen%27s_kappa
- `elasticsearch-relevance-tests` -
  consumes judgment lists for `_rank_eval` (Elasticsearch and
  OpenSearch) and the Solr nDCG harness
- `vector-search-recall-tests` -
  consumes judgment lists for recall@k evaluation