Redial Movie Recommendation Eval

---
name: redial-movie-recommendation-eval
description: Evaluates modular components of a conversational recommendation system, specifically cold-start movie rating prediction and movie opinion sentiment analysis (seen/liked status) from dialogue text. Use when the user wants to benchmark on REDIAL, MovieLens, or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1812.07617
  bibtex_key: li2018towards
  confidence: high
---

# redial-movie-recommendation-eval

> Towards Deep Conversational Recommendations — Li et al. (2018) (arXiv:1812.07617, 2018)

## What this evaluates

Evaluates modular components of a conversational recommendation system, specifically cold-start movie rating prediction and movie opinion sentiment analysis (seen/liked status) from dialogue text.

## Datasets

- **REDIAL** — total 10000; splits: validation (-1), test (-1)
- **MovieLens** — total 26000000; splits: train (-1), validation (-1), test (-1)

## Metrics

- `Cohen's kappa` — range: [-1, 1]
  - Measures inter-rater agreement between true labels and predictions, correcting for chance. κ = (p_o - p_e) / (1 - p_e).
- `RMSE` **(primary)** — range: [0, 1] or [0, 5]
  - Root Mean Squared Error between predicted and actual ratings. RMSE = sqrt(mean((y_true - y_pred)^2)).

## Input / output format

**Input**: Sentiment: text spans between first and last mention of a movie in a dialogue turn. Recommendation: binary or scaled ratings from a conversation's history (treated as a user profile).

**Output**: Sentiment: binary classification labels (seen/not seen, liked/not liked). Recommendation: predicted rating value (0/1 or 0.5–5).

## Scoring recipe

```python
def compute_kappa(y_true, y_pred):
    return kappa_score(y_true, y_pred)

def compute_rmse(y_true, y_pred):
    return np.sqrt(np.mean((np.array(y_true) - np.array(y_pred)) ** 2))
```

## Common pitfalls

- Class imbalance in sentiment labels (81% liked) requires explicit loss weighting during training.
- Cold-start evaluation treats each conversation as an independent user, ignoring cross-conversation user history.
- MovieLens ratings must be binarized (threshold ≥2) to match REDIAL's distribution before pre-training.

## Evidence (verbatim from paper)

> We also provide Cohen's kappa coefficient [26] for each model and prediction task. Cohen's kappa measures the agreement between the true label and the predictions. ... Table 2 shows the RMSE obtained on the test set. ... Randomly chosen user-item ratings are held out for validation and test, and only training ratings are used as inputs. Following Sedhain et al. [1], we sampled the training, validation, and test set in a 80-10-10 proportion, and repeated this splitting procedure five times, reporting the average RMSE.

## Citation

```bibtex
@misc{li2018towards,
  title={Towards Deep Conversational Recommendations},
  author={Li et al. (2018)},
  year={2018},
  note={arXiv:1812.07617}
}
```

- arXiv: 1812.07617