Dualrec Movielens Eval

---
name: dualrec-movielens-eval
description: Evaluates a hybrid sequential and LLM-based framework for next-item movie recommendation. It probes the model's ability to capture temporal user preferences and semantic genre consistency to predict the next movie a user will watch. Use when the user wants to benchmark on MovieLens-1M, or asks about evaluating this task. Reports NDCG@5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.13957
  bibtex_key: li2025dualrec
  confidence: high
---

# dualrec-movielens-eval

> DUALRec: A Hybrid Sequential and Language Model Framework for Context-Aware Movie Recommendation — Li et al. (2025) (arXiv:2507.13957, 2025)

## What this evaluates

Evaluates a hybrid sequential and LLM-based framework for next-item movie recommendation. It probes the model's ability to capture temporal user preferences and semantic genre consistency to predict the next movie a user will watch.

## Datasets

- **MovieLens-1M** — total ?; splits: test (-1)

## Metrics

- `HR@1` — range: [0, 1]
  - Proportion of test cases where the relevant item appears within the top-1 recommendations. Calculated as the number of hits in top-k divided by the total number of test cases.
- `HR@5` — range: [0, 1]
  - Proportion of test cases where the relevant item appears within the top-5 recommendations. Calculated as the number of hits in top-k divided by the total number of test cases.
- `NDCG@1` — range: [0, 1]
  - Evaluates ranking quality by giving higher weight to correct predictions that appear earlier in the recommendation list, normalized by the ideal DCG.
- `NDCG@5` **(primary)** — range: [0, 1]
  - Evaluates ranking quality by giving higher weight to correct predictions that appear earlier in the recommendation list, normalized by the ideal DCG.
- `Genre Jaccard Similarity` — range: [0, 1]
  - Computes the Jaccard similarity between the genre tags of the top-1 recommended movie and the user's actual next watched movie to quantify semantic relevance.

## Input / output format

**Input**: User's sequential interaction history and temporal signals, formatted as a natural language prompt for the LLM component, alongside sequential features for the LSTM component.

**Output**: Top-k ranked list of recommended movies (k=1 for Genre Jaccard, k=5 for HR/NDCG).

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, k=5):
    hits = 0
    ndcg_sum = 0.0
    jaccard_sum = 0.0
    for pred_list, true_item in zip(predictions, ground_truth):
        if true_item in pred_list[:k]:
            hits += 1
        dcg = sum(1.0 / log2(i + 2) for i, item in enumerate(pred_list[:k]) if item == true_item)
        idcg = 1.0 / log2(2)
        ndcg_sum += dcg / idcg
        rec_genres = get_genres(pred_list[0])
        true_genres = get_genres(true_item)
        jaccard_sum += len(rec_genres & true_genres) / len(rec_genres | true_genres)
    return hits / len(predictions), ndcg_sum / len(predictions), jaccard_sum / len(predictions)
```

## Common pitfalls

- Confusing the hard prompt format used in this study with the numerical soft prompts used in the baseline (Xu et al., 2025).
- Assuming Genre Jaccard Similarity is computed over the full top-k list instead of strictly the top-1 recommendation as specified.
- Overlooking that the evaluation constructs natural language prompts from user activity for the LLM, which differs from pure sequential model baselines.

## Evidence (verbatim from paper)

> To assess the performance of the recommendation model, we employ standard metrics that are often used to assess recommender models (Xu et al., 2025; Wang & Lim, 2023), including Hit Rate at n (HR@(1,5)) and Normalized Discounted Cumulative Gain at n (NDCG@(1,5)). HR@n measures the proportion of test cases where the relevant item appears within the top-n recommendations, calculated as the number of hits in top-k divided by the total number of test cases. NDCG@n evaluates the ranking quality by giving higher weight to correct predictions that appear earlier in the recommendation list.

In addition to ranking accuracy metrics, we also use Genre Jaccard Similarity to assess the semantic understanding of the model’s recommendations. This metric compares the overlap of genre tags between the recommended items and the actual items the user watched next. By doing so, we evaluate how well each DUALRec variant preserves genre-level consistency, providing a complementary perspective on whether the generated movie recommendations align with the thematic preferences reflected in the user’s true viewing behavior. Specifically, we compute the Jaccard similarity between the top-1 recommended movie

## Citation

```bibtex
@misc{li2025dualrec,
  title={DUALRec: A Hybrid Sequential and Language Model Framework for Context-Aware Movie Recommendation},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2507.13957}
}
```

- arXiv: 2507.13957