Vimrhp Eval

---
name: vimrhp-eval
description: Evaluates a model's ability to predict the helpfulness of product reviews by jointly processing textual descriptions and visual content. It probes multimodal alignment and ranking capabilities in low-resource language settings, specifically Vietnamese. Use when the user wants to benchmark on ViMRHP, or asks about evaluating this task. Reports NDCG@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.07416
  bibtex_key: nguyen2025vimrhp
  confidence: high
---

# vimrhp-eval

> ViMRHP: A Vietnamese Benchmark Dataset for Multimodal Review Helpfulness Prediction via Human-AI Collaborative Annotation — Nguyen et al. (2025) (arXiv:2505.07416, 2025)

## What this evaluates

Evaluates a model's ability to predict the helpfulness of product reviews by jointly processing textual descriptions and visual content. It probes multimodal alignment and ranking capabilities in low-resource language settings, specifically Vietnamese.

## Datasets

- **ViMRHP** — total 46000; splits: unspecified (-1); repo https://github.com/trng28/ViMRHP

## Metrics

- `MAP` — range: [0, 1]
  - Mean Average Precision: computes the average precision across all queries, averaging precision values at ranks where relevant items occur.
- `NDCG@K` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff K: measures ranking quality by discounting gains logarithmically with rank position and normalizing by the ideal DCG. Evaluated at K ∈ {1, 3, 5}.

## Input / output format

**Input**: Product description, review text, and associated product/review images.

**Output**: A predicted helpfulness score or ranking score per review.

## Scoring recipe

```python
def compute_ndcg_at_k(pred_scores, true_labels, k):
    ranked_idx = np.argsort(-pred_scores)
    true_ranked = [true_labels[i] for i in ranked_idx[:k]]
    dcg = sum(l / np.log2(i + 2) for i, l in enumerate(true_ranked))
    ideal = sorted(true_labels, reverse=True)[:k]
    idcg = sum(l / np.log2(i + 2) for i, l in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0

def compute_map(pred_scores, true_labels):
    precisions = []
    rel_count = 0
    for i, score in enumerate(np.argsort(-pred_scores)):
        if true_labels[score] > 0:
            rel_count += 1
            precisions.append(rel_count / (i + 1))
    return np.mean(precisions) if precisions else 0.0
```

## Common pitfalls

- Evaluating at K values other than 1, 3, or 5, which the authors explicitly tie to user reading behavior.
- Ignoring the multimodal requirement by using text-only baselines or dropping image features, despite the dataset's core contribution.
- Failing to apply the MatchZoo threshold of 3.0 when computing MAP/NDCG, which affects how relevance is binarized or graded.

## Evidence (verbatim from paper)

> Following the study by Liu et al. [[4]] on the Amazon-MRHP and Lazada-MRHP datasets, we adopt two evaluation metrics frequently used in recommendation: MAP (Mean Average Precision) and NDCG@K (Normalized Discounted Cumulative Gain), where $K\in{1,3,5}$, to assess the ViMRHP dataset.

## Citation

```bibtex
@misc{nguyen2025vimrhp,
  title={ViMRHP: A Vietnamese Benchmark Dataset for Multimodal Review Helpfulness Prediction via Human-AI Collaborative Annotation},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2505.07416}
}
```

- arXiv: 2505.07416