Helm Recommender Eval

---
name: helm-recommender-eval
description: Evaluates LLM-powered recommender systems across five human-centered dimensions: intent alignment, explanation quality, interaction naturalness, trust & transparency, and fairness & diversity. It combines expert human ratings with automated accuracy and fairness metrics across multiple domains and interaction scenarios. Use when the user wants to benchmark on MovieLens-1M, Amazon Books, Yelp, or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.19197
  bibtex_key: mehta2026helm
  confidence: high
---

# helm-recommender-eval

> HELM: A Human-Centered Evaluation Framework for LLM-Powered Recommender Systems — Mehta (2026) (arXiv:2601.19197, 2026)

## What this evaluates

Evaluates LLM-powered recommender systems across five human-centered dimensions: intent alignment, explanation quality, interaction naturalness, trust & transparency, and fairness & diversity. It combines expert human ratings with automated accuracy and fairness metrics across multiple domains and interaction scenarios.

## Datasets

- **MovieLens-1M** — total 1000000; splits: test (-1)
- **Amazon Books** — total 287650; splits: test (-1)
- **Yelp** — total 612840; splits: test (-1)

## Metrics

- `Hit Rate@10` — range: [0, 1]
  - Fraction of ground-truth relevant items appearing in the top-10 recommendations.
- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10, measuring ranking quality with logarithmic position discounting.
- `Explanation Faithfulness` — range: [0, 1]
  - Proportion of claimed item attributes in the LLM's explanation that can be verified against ground-truth metadata.
- `Response Consistency` — range: [0, 1]
  - Cosine similarity between model responses to paraphrased versions of the same query.
- `Gini Coefficient` — range: [0, 1]
  - Measures inequality in the frequency of recommended items across all scenarios; higher values indicate stronger popularity bias.
- `Coverage@100` — range: [0, 1]
  - Proportion of unique catalog items that appear in the top-100 recommendations across all scenarios.
- `Intra-List Diversity (ILD)` — range: [0, 1]
  - Average pairwise dissimilarity between items within each recommendation list.

## Input / output format

**Input**: Scripted user profile, interaction history, and scenario context (e.g., cold-start, preference refinement, contextual request) provided as prompts to the recommender system.

**Output**: Ranked list of recommended items (top-10 or top-100) accompanied by natural language explanations and multi-turn conversational responses.

## Scoring recipe

```python
def compute_metrics(predictions, gold, metadata, all_items):
    top10 = predictions[:10]
    hr = sum(1 for x in top10 if x in gold) / len(gold)
    dcg = sum(1/math.log2(i+2) for i, x in enumerate(top10) if x in gold)
    idcg = sum(1/math.log2(i+2) for i in range(len(gold)))
    ndcg = dcg/idcg if idcg > 0 else 0
    faithfulness = sum(1 for a in claimed_attrs if a in metadata) / len(claimed_attrs)
    consistency = cosine_sim(response, paraphrased_response)
    counts = Counter(predictions)
    gini = 1 - (2 * sum(counts.values() * (len(counts) - np.arange(len(counts)) - 1)) / (len(counts) * sum(counts.values())))
    coverage = len(set(predictions[:100])) / len(all_items)
    ild = np.mean([dissimilarity(i, j) for i, j in combinations(predictions[:100], 2)])
    return {'HR@10': hr, 'NDCG@10': ndcg, 'Faithfulness': faithfulness, 'Consistency': consistency, 'Gini': gini, 'Coverage@100': coverage, 'ILD': ild}
```

## Common pitfalls

- Expert Likert scores (5-point scale) are aggregated per scenario but not combined into a single composite score, making cross-dimension comparison difficult.
- Explanation Faithfulness relies on external metadata availability; missing metadata can artificially lower faithfulness scores regardless of model capability.
- Gini Coefficient measures popularity bias across the entire evaluation set, not per-user fairness, which may misrepresent individual recommendation quality.

## Evidence (verbatim from paper)

> Complementing expert evaluation, we compute automated metrics: Hit Rate@10 and NDCG@10: Traditional accuracy metrics for baseline comparison. Explanation Faithfulness: Proportion of claimed item attributes verifiable against metadata. Response Consistency: Cosine similarity between responses to paraphrased queries. Gini Coefficient: Inequality in item recommendation frequency. Coverage@100: Proportion of catalog items appearing in the top-100 recommendations. Intra-List Diversity (ILD): Average pairwise dissimilarity within recommendation lists.

## Citation

```bibtex
@misc{mehta2026helm,
  title={HELM: A Human-Centered Evaluation Framework for LLM-Powered Recommender Systems},
  author={Mehta (2026)},
  year={2026},
  note={arXiv:2601.19197}
}
```

- arXiv: 2601.19197