Harrison Eval

---
name: harrison-eval
description: This benchmark evaluates a model's ability to recommend relevant hashtags for real-world social media images using only visual input. It probes contextual image understanding and multi-label classification by measuring how well predicted hashtags align with actual user-generated tags. Use when the user wants to benchmark on HARRISON, or asks about evaluating this task. Reports Precision@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1605.05054
  bibtex_key: [REDACTED]
  confidence: high
---

# harrison-eval

> HARRISON: A Benchmark on HAshtag Recommendation for Real-world Images in Social Networks — Park et al. (2016) (arXiv:1605.05054, 2016)

## What this evaluates

This benchmark evaluates a model's ability to recommend relevant hashtags for real-world social media images using only visual input. It probes contextual image understanding and multi-label classification by measuring how well predicted hashtags align with actual user-generated tags.

## Datasets

- **HARRISON** — total 57383; splits: test (-1)

## Metrics

- `Precision@1` **(primary)** — range: percent
  - The fraction of hashtags in the top 1 predicted result that exactly match the ground truth set. Formula: |Result(1) ∩ GT| / |Result(1)|.
- `Recall@5` — range: percent
  - The fraction of ground truth hashtags covered by the top 5 predicted results. Formula: |Result(5) ∩ GT| / |GT|.
- `Accuracy@5` — range: percent
  - A binary indicator that is 1 if at least one predicted hashtag in the top 5 matches any ground truth hashtag, and 0 otherwise.

## Input / output format

**Input**: Single real-world social media image (processed via pre-trained VGG-16 visual features).

**Output**: Ranked list of predicted hashtags (top K, where K=1 for precision and K=5 for recall/accuracy).

## Scoring recipe

```python
def compute_metrics(preds, gold, k_prec=1, k_rec=5, k_acc=5):
    top_k_prec = set(preds[:k_prec])
    top_k_rec = set(preds[:k_rec])
    gt = set(gold)
    inter_prec = top_k_prec & gt
    inter_rec = top_k_rec & gt
    prec = len(inter_prec) / len(top_k_prec) if top_k_prec else 0.0
    rec = len(inter_rec) / len(gt) if gt else 0.0
    acc = 1.0 if inter_rec else 0.0
    return {'precision@1': prec, 'recall@5': rec, 'accuracy@5': acc}
```

## Common pitfalls

- K is metric-specific: Precision uses K=1, while Recall and Accuracy use K=5. Using a uniform K across all metrics will yield incorrect scores.
- Evaluation relies on exact string matching between predicted and ground truth hashtags. Semantic similarity, lemmatization, or synonym matching is not applied during scoring.
- The protocol treats hashtags as independent labels, ignoring co-occurrence dependencies or hierarchical relationships between tags.

## Evidence (verbatim from paper)

> In our experiments, we set K to 1 for precision and 5 for recall and accuracy considering the average number of associated hashtags per image in the HARRISON dataset. We evaluated the baseline results on Precision@1, Recall@5, and Accuracy@5 by averaging over all images in the test set.

## Citation

```bibtex
@misc{park2016harrison,
  title={HARRISON: A Benchmark on HAshtag Recommendation for Real-world Images in Social Networks},
  author={Park et al. (2016)},
  year={2016},
  note={arXiv:1605.05054}
}
```

- arXiv: 1605.05054