News Rec Eval

---
name: news-rec-eval
description: Evaluates the classification and ranking performance of LLM-based versus deep learning-based news recommendation models. It also measures the diversity of recommended items and how well the recommendations align with individual user history (personalization). Use when the user wants to benchmark on MIND-small, Adressa (one-week), or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.09797
  bibtex_key: [REDACTED]
  confidence: high
---

# news-rec-eval

> A Survey on LLM-based News Recommender Systems — Wang et al. (2025) (arXiv:2502.09797, 2025)

## What this evaluates

Evaluates the classification and ranking performance of LLM-based versus deep learning-based news recommendation models. It also measures the diversity of recommended items and how well the recommendations align with individual user history (personalization).

## Datasets

- **MIND-small** — total ?; splits: (unstated)
- **Adressa (one-week)** — total ?; splits: (unstated)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the ROC Curve. Calculated as the fraction of positive-negative sample pairs where the predicted score for the positive sample exceeds that of the negative sample: AUC = (1/|P||N|) * sum_{p in P, n in N} I(s(p) > s(n)).
- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank. Average of the reciprocal of the rank position of the first relevant item across queries: MRR = (1/|Q|) * sum_{i=1}^{|Q|} (1/Rank_i).
- `nDCG@k` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at k. Ratio of the DCG of the predicted ranking to the ideal DCG: nDCG@k = sum_{i=1}^k (rel_i / log2(i+1)) / sum_{i=1}^k (rel*_i / log2(i+1)).
- `Recall@k` — range: [0, 1]
  - Fraction of actual positive samples correctly retrieved in the top-k list: Recall = TP / (TP + FN).
- `Precision@k` — range: [0, 1]
  - Fraction of retrieved items in the top-k list that are relevant: Precision = TP / (TP + FP).
- `HR@k` — range: [0, 1]
  - Hit Rate at k. Proportion of users for whom at least one relevant item appears in the top-k recommendations: HR@k = (1/|U|) * sum_{u in U} I(R_u intersect R_hat_u(k) != emptyset).
- `Diversity@A_p@k` — range: [0, 1]
  - Aspect-based diversity. Normalized entropy over aspect distributions in the top-k list: D_Ap@k = -sum_{j in A_p} (p(j) log p(j) / log(|A_p|)).
- `Personalization@A_p@k` — range: [0, 1]
  - Jaccard similarity between user history and recommendation list aspect probabilities: PS_Ap@k = sum_j min(R_j, H_j) / sum_j max(R_j, H_j).

## Input / output format

**Input**: User interaction logs (IDs, clicks, impressions, demographics like region/time/browser) paired with news metadata (titles, abstracts, categories, entities, bodies).

**Output**: Predicted relevance scores for candidate news items, or a top-k ranked list of recommended news.

## Scoring recipe

```python
def compute_metrics(pred_scores, true_labels, k=5):
    ranked = np.argsort(pred_scores)[::-1][:k]
    hits = np.isin(ranked, np.where(true_labels)[0])
    hr = hits.mean()
    tp = hits.sum()
    precision = tp / k
    recall = tp / max(true_labels.sum(), 1)
    # MRR: 1/rank of first hit if any, else 0
    mrr = 1.0 / (np.where(hits)[0][0] + 1) if hits.any() else 0.0
    # nDCG@k: DCG / IDCG
    dcg = np.sum(hits / np.log2(np.arange(2, k+2)))
    idcg = np.sum(np.sort(true_labels)[:k][::-1] / np.log2(np.arange(2, k+2)))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    # AUC: pair-wise comparison
    auc = np.mean([1 if pred_scores[p] > pred_scores[n] else 0 for p in np.where(true_labels)[0] for n in np.where(~true_labels)[0]])
    return {'AUC': auc, 'MRR': mrr, 'nDCG@k': ndcg, 'Recall@k': recall, 'Precision@k': precision, 'HR@k': hr}
```

## Common pitfalls

- Only MIND-small and one-week Adressa subsets are used due to resource constraints, which may not reflect full-scale performance.
- Conflicting results on Adressa arise from LLMs' difficulty with multilingual news encoding compared to DL baselines, not necessarily model architecture flaws.
- Negative sampling strategy for AUC and tie-breaking rules for ranking metrics are not explicitly defined in the protocol.

## Evidence (verbatim from paper)

> In our experiments, we compute top 5 and 10 scores in terms of nDCG, Hit, Recall, Precision, diversity, and personalization.

## Citation

```bibtex
@misc{wang2025surveyllmnewsrec,
  title={A Survey on LLM-based News Recommender Systems},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2502.09797}
}
```

- arXiv: 2502.09797