Multimodal Rec Benchmark

---
name: multimodal-rec-benchmark
description: Evaluates the performance of multimodal deep learning recommender systems across five Amazon product categories. It probes both standard recommendation accuracy and beyond-accuracy dimensions such as novelty, diversity, popularity bias, and catalog coverage. Use when the user wants to benchmark on Amazon (Office, Toys, Beauty, Sports, Clothing), or asks about evaluating this task. Reports Recall@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.05273
  bibtex_key: malitesta2023formalizing
  confidence: high
---

# multimodal-rec-benchmark

> Formalizing Multimedia Recommendation through Multimodal Deep Learning — Malitesta et al. (2023) (arXiv:2309.05273, 2023)

## What this evaluates

Evaluates the performance of multimodal deep learning recommender systems across five Amazon product categories. It probes both standard recommendation accuracy and beyond-accuracy dimensions such as novelty, diversity, popularity bias, and catalog coverage.

## Datasets

- **Amazon (Office, Toys, Beauty, Sports, Clothing)** — total ?; splits: train (-1), test (-1)

## Metrics

- `Recall@k` **(primary)** — range: [0, 1]
  - Fraction of relevant items retrieved in the top-k list per user, averaged over all users: (1/|U|) * sum_u(|Rel_u@k| / |Rel_u|).
- `nDCG@k` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at k. Averages the ratio of DCG@k to ideal DCG@k over users, where DCG@k sums (2^rel-1)/log2(i+1) for ranked positions.
- `EFD@k` — range: [0, 1]
  - Expected Free Discovery at k. Quantifies novelty by weighting predicted relevance with the inverse collection frequency (-log2 p(i|seen, theta)) to measure long-tail item exposure.
- `Gini@k` — range: [0, 1]
  - Normalized Gini index at k. Measures popularity disparity in top-k lists; higher values indicate wider item diversity, lower values indicate concentration on popular items.
- `APLT@k` — range: [0, 1]
  - Average Percentage of Long-tail items at k. Calculates the percentage of niche (long-tail) items in the top-k recommendations, averaged over users.
- `iCov@k` — range: percent
  - Item coverage at k. Percentage of training catalog items that appear in any user's top-k recommendation list: |union_u(I_hat_u@k)| / |I_train| * 100%.

## Input / output format

**Input**: User-item interaction history with timestamps, plus pre-extracted 4,096-dimensional visual embeddings and 1,024-dimensional textual embeddings for each item.

**Output**: A ranked list of top-k items for each user.

## Scoring recipe

```python
def compute_metrics(predictions, gold, k=20):
    recalls, ndcgs = [], []
    for u in gold:
        rel = gold[u]
        pred_k = predictions[u][:k]
        recalls.append(len(set(pred_k) & rel) / len(rel) if rel else 0)
        # nDCG follows standard DCG/IDCG ratio over ranked positions
    return {'Recall@k': sum(recalls)/len(recalls), 'nDCG@k': sum(ndcgs)/len(ndcgs)}
```

## Common pitfalls

- Relying solely on accuracy metrics (Recall/nDCG) overlooks critical beyond-accuracy dimensions like novelty, diversity, and popularity bias.
- The validation split is created by removing 50% of the test set, which is a non-standard hold-out procedure that may affect hyperparameter selection fairness.
- Visual and textual features are pre-extracted embeddings rather than learned jointly from raw data, limiting direct comparison with end-to-end multimodal models.

## Evidence (verbatim from paper)

> For the recommendation accuracy, we consider the Recall@k and the nDCG@k; for the novelty and diversity, we measure the EFD@k and the Gini@k, respectively; for the popularity bias, we calculate the APTL@k; finally, as a general index of how recommendations cover the entire catalog of products, we adopt the iCov@k.

## Citation

```bibtex
@misc{malitesta2023formalizing,
  title={Formalizing Multimedia Recommendation through Multimodal Deep Learning},
  author={Malitesta et al. (2023)},
  year={2023},
  note={arXiv:2309.05273}
}
```

- arXiv: 2309.05273