Yelp Rating Prediction Eval

---
name: yelp-rating-prediction-eval
description: Predicts a 1-to-5 star rating for a restaurant review based on its text content. It probes a model's ability to capture sentiment, domain-specific linguistic patterns, and fine-grained textual features for multi-class classification. Use when the user wants to benchmark on Yelp Dataset, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1605.05362
  bibtex_key: asghar2016yelp
  confidence: high
---

# yelp-rating-prediction-eval

> Yelp Dataset Challenge: Review Rating Prediction — Asghar (2016) (arXiv:1605.05362, 2016)

## What this evaluates

Predicts a 1-to-5 star rating for a restaurant review based on its text content. It probes a model's ability to capture sentiment, domain-specific linguistic patterns, and fine-grained textual features for multi-class classification.

## Datasets

- **Yelp Dataset** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of reviews where the predicted star rating matches the ground truth rating.
- `RMSE` — range: other
  - Square root of the average squared difference between predicted and actual star ratings.

## Input / output format

**Input**: Preprocessed text review (lowercased, punctuation and stop words removed).

**Output**: Predicted star rating as an integer class label from {1, 2, 3, 4, 5}.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    n = len(gold)
    accuracy = sum(1 for p, g in zip(preds, gold) if p == g) / n
    rmse = (sum((p - g) ** 2 for p, g in zip(preds, gold)) / n) ** 0.5
    return {'accuracy': accuracy, 'rmse': rmse}
```

## Common pitfalls

- The paper performs 3-fold cross-validation only on the 80% training set for hyperparameter tuning (e.g., SVM C parameter), not for final evaluation on the test set.
- Feature extraction (n-gram dictionaries, TF-IDF, LSI matrices) must be fitted exclusively on the training data to prevent data leakage when scoring the test set.
- Preprocessing steps (lowercasing, stop-word removal, punctuation stripping) are explicitly required before feature extraction and must be applied identically to test reviews.

## Evidence (verbatim from paper)

> We use 80% of the dataset for training, and 20% for testing. For each of the sixteen prediction systems, we perform 3-fold cross validation on the training set and compute two metrics, Root Mean Squared Error (RMSE) and accuracy, for the training fold as well as the validation fold.

## Citation

```bibtex
@misc{asghar2016yelp,
  title={Yelp Dataset Challenge: Review Rating Prediction},
  author={Asghar (2016)},
  year={2016},
  note={arXiv:1605.05362}
}
```

- arXiv: 1605.05362