Graphrec Eval

---
name: graphrec-eval
description: Evaluates the predictive accuracy of social recommendation models by forecasting user-item ratings. It jointly leverages user-item interaction graphs and user-user social graphs to learn co-embeddings, testing the model's ability to integrate heterogeneous social tie strengths and opinion signals into rating prediction. Use when the user wants to benchmark on Ciao, Epinions, or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1902.07243
  bibtex_key: fan2019graphrec
  confidence: high
---

# graphrec-eval

> Graph Neural Networks for Social Recommendation — Fan et al. (2019) (arXiv:1902.07243, 2019)

## What this evaluates

Evaluates the predictive accuracy of social recommendation models by forecasting user-item ratings. It jointly leverages user-item interaction graphs and user-user social graphs to learn co-embeddings, testing the model's ability to integrate heterogeneous social tie strengths and opinion signals into rating prediction.

## Datasets

- **Ciao** — total 283319; splits: train (-1), val (-1), test (-1); repo https://github.com/wenqifan03/GraphRec-WWW19
- **Epinions** — total 764352; splits: train (-1), val (-1), test (-1); repo https://github.com/wenqifan03/GraphRec-WWW19

## Metrics

- `MAE` — range: other
  - Mean Absolute Error: (1/N) Σ |r_ui - r̂_ui|. Lower values indicate better predictive accuracy.
- `RMSE` **(primary)** — range: other
  - Root Mean Square Error: √((1/N) Σ (r_ui - r̂_ui)²). Lower values indicate better predictive accuracy.

## Input / output format

**Input**: User ID, Item ID, social graph adjacency, and opinion ratings.

**Output**: Predicted continuous rating score.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    n = len(gold)
    mae = sum(abs(p - g) for p, g in zip(preds, gold)) / n
    rmse = (sum((p - g)**2 for p, g in zip(preds, gold)) / n) ** 0.5
    return {'MAE': mae, 'RMSE': rmse}
```

## Common pitfalls

- Metrics are error-based (lower is better), not accuracy-based.
- Evaluation splits vary by training ratio (60% or 80%), which changes the test set size and may affect comparability.
- Some baselines (e.g., GCMC+SN) preprocess social graphs with node2vec before feeding to the model, which differs from using raw adjacency matrices.

## Evidence (verbatim from paper)

> In order to evaluate the quality of the recommendation algorithms, two popular metrics are adopted to evaluate the predictive accuracy, namely Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). Smaller values of MAE and RMSE indicate better predictive accuracy. Note that small improvement in RMSE or MAE terms can have a significant impact on the quality of the top-few recommendations. For each dataset, we used x% as a training set to learning parameters, (1-x%)/2 as a validation set to tune hyper-parameters, and (1-x%)/2 as a testing set for the final performance comparison, where x was varied as {80%,60%}.

## Citation

```bibtex
@misc{fan2019graphrec,
  title={Graph Neural Networks for Social Recommendation},
  author={Fan et al. (2019)},
  year={2019},
  note={arXiv:1902.07243}
}
```

- arXiv: 1902.07243