Mm Graph Eval

---
name: mm-graph-eval
description: Evaluates multimodal graph learning models on node classification, link prediction, and knowledge graph completion tasks. It probes the ability of models to integrate high-resolution visual and textual node features with graph structure to make accurate predictions across diverse real-world domains. Use when the user wants to benchmark on Amazon-Sports, Amazon-Cloth, Goodreads-LP, Goodreads-NC, Ele-fashion, MM-CoDEx-s, MM-CoDEx-m, or asks about evaluating this task. Reports MRR, accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.16321
  bibtex_key: zhu2024mmgraph
  confidence: high
---

# mm-graph-eval

> Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learning — Zhu et al. (2024) (arXiv:2406.16321, 2024)

## What this evaluates

Evaluates multimodal graph learning models on node classification, link prediction, and knowledge graph completion tasks. It probes the ability of models to integrate high-resolution visual and textual node features with graph structure to make accurate predictions across diverse real-world domains.

## Datasets

- **Amazon-Sports** — total 50250; splits: train (-1), valid (-1), test (-1)
- **Amazon-Cloth** — total 125839; splits: train (-1), valid (-1), test (-1)
- **Goodreads-LP** — total 636502; splits: train (-1), valid (-1), test (-1)
- **Goodreads-NC** — total 685294; splits: train (-1), valid (-1), test (-1)
- **Ele-fashion** — total 97766; splits: train (-1), valid (-1), test (-1)
- **MM-CoDEx-s** — total 1383; splits: train (14298), valid (784), test (802)
- **MM-CoDEx-m** — total 7697; splits: train (47617), valid (2628), test (2595)

## Metrics

- `MRR` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank: the average of the reciprocal of the rank of the correct edge or triple among all ranked candidates.
- `Hits@10` — range: [0, 1]
  - Fraction of positive edges or triples ranked within the top 10 candidates.
- `Hits@1` — range: [0, 1]
  - Fraction of positive edges or triples ranked as the top candidate.
- `Hits@3` — range: [0, 1]
  - Fraction of positive edges or triples ranked within the top 3 candidates.
- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted node categories out of the total number of nodes.

## Input / output format

**Input**: Graph-structured data where each node contains textual features (e.g., product titles, book descriptions, Wikipedia text) and visual features (e.g., product images, book covers, entity images), along with adjacency information.

**Output**: For link prediction and KGC: a ranked list of candidate edges or triples. For node classification: a predicted category label from a fixed set (10 categories for Goodreads-NC, 12 for Ele-fashion).

## Scoring recipe

```python
def compute_mrr(ranks):
    return sum(1.0 / r for r in ranks) / len(ranks)

def compute_hits_at_k(ranks, k):
    return sum(1 for r in ranks if r <= k) / len(ranks)

def compute_accuracy(predictions, labels):
    return sum(1 for p, l in zip(predictions, labels) if p == l) / len(labels)
```

## Common pitfalls

- Failing to explicitly remove validation and test edges from the graph during training, which causes information leakage.
- Using random negative sampling instead of hard negatives (e.g., HeaRT) for link prediction, which underestimates model difficulty and inflates scores.
- Not filtering out entities/nodes lacking multimodal features before splitting, which can introduce distribution shifts across train/val/test sets.

## Evidence (verbatim from paper)

> For evaluation metrics, we report MRR, Hits@10, and Hits@1, the three most commonly-used evaluation metrics for link prediction [16, 22].

## Citation

```bibtex
@misc{zhu2024mmgraph,
  title={Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learning},
  author={Zhu et al. (2024)},
  year={2024},
  note={arXiv:2406.16321}
}
```

- arXiv: 2406.16321