Trec Rts Metrics

---
name: trec-rts-metrics
description: Evaluates real-time tweet summarization systems by measuring their ability to push relevant, non-redundant tweets within fixed temporal windows, while penalizing system latency and irrelevant outputs. Use when the user has predictions and gold and needs to compute Expected Gain (EG).
metadata:
  skill_kind: metric
  source_arxiv: 1712.04671
  bibtex_key: hubert2017trecrts
  confidence: high
---

# trec-rts-metrics

> Everything You Always Wanted to Know About TREC RTS* (*But Were Afraid to Ask) — Hubert et al. (2017) (arXiv:1712.04671, 2017)

## What this evaluates

Evaluates real-time tweet summarization systems by measuring their ability to push relevant, non-redundant tweets within fixed temporal windows, while penalizing system latency and irrelevant outputs.

## Datasets

- **TREC RTS Track Data** — total ?; splits: test (-1)

## Metrics

- `Expected Gain (EG)` **(primary)** — range: [0, 1]
  - EG(w_j, S_i) = (1 / |T_i(w_j)|) * Σ g(t), where g(t)=1 if t is the first relevant tweet in its cluster, else 0. Variants handle silent days: EG-0 gives 0, EG-1 gives 1 if no tweets pushed, EG-p gives (N - non_rel)/N.
- `Normalized Cumulative Gain (nCG)` — range: [0, 1]
  - nCG(w_j, S_i) = (1 / Z) * G(S_i, w_j), where Z is the maximum possible gain given the N tweets/day limit. Shares EG-0/1/p silent day variants.
- `Gain Minus Pain (GMP)` — range: other
  - GMP = α * ΣG - (1-α) * P, where P is the count of non-relevant tweets returned. α is tuned to 0.33, 0.50, or 0.66.
- `Latency` — range: other
  - Sum over relevant tweets of (push_time - creation_time) for the oldest tweet pushed per cluster.

## Input / output format

**Input**: For each temporal window and interest profile: a set of candidate tweets with creation timestamps, pre-defined semantic clusters, and the system's output list of pushed tweets with push timestamps.

**Output**: A sequence of pushed tweets per window, ordered by push time, with associated timestamps.

## Scoring recipe

```python
def compute_eg(pushed, window_tweets, N, variant="EG-1"):
    gain = 0
    seen_clusters = set()
    for t in window_tweets:
        if t.cluster not in seen_clusters:
            if t in pushed:
                gain += 1
                seen_clusters.add(t.cluster)
    n_pushed = len(pushed)
    if variant == "EG-0":
        return gain / n_pushed if n_pushed > 0 else 0
    elif variant == "EG-1":
        return 1.0 if n_pushed == 0 else gain / n_pushed
    else: # EG-p
        non_rel = n_pushed - gain
        return (N - non_rel) / N
```

## Common pitfalls

- Relevance is system-dependent: a tweet is only relevant if it is the first one retrieved from its cluster by that specific system.
- Silent day handling variants (EG-0, EG-1, EG-p) drastically change scores and should not be mixed.
- Gain is calculated based on tweet publication time, not push time, so systems can score in windows where they returned nothing.
- The Latency metric can yield a perfect score without returning any relevant tweets.

## Evidence (verbatim from paper)

> The expected gain metric, denoted by EG, is adapted from [2]. Given a time window w_j, it is evaluated as: EG(w_j, S_i) = (1 / |T_i(w_j)|) * G(S_i, w_j) where |T_i(w_j)| is the number of tweets returned by S_i and published during w_j.

## Citation

```bibtex
@misc{hubert2017trecrts,
  title={Everything You Always Wanted to Know About TREC RTS* (*But Were Afraid to Ask)},
  author={Hubert et al. (2017)},
  year={2017},
  note={arXiv:1712.04671}
}
```

- arXiv: 1712.04671