Hit Ratio Eval

---
name: hit-ratio-eval
description: Evaluates the ability of LLM-based sequential recommendation models to predict the next item in a user's interaction history. It specifically probes how well models capture temporal dynamics by incorporating irregular time intervals between interactions, and assesses performance under warm and cold-start conditions. Use when the user wants to benchmark on Amazon Reviews (Video Games, CDs and Vinyl, Books), or asks about evaluating this task. Reports Hit Ratio@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.23209
  bibtex_key: du2025interval
  confidence: high
---

# hit-ratio-eval

> Not Just What, But When: Integrating Irregular Intervals to LLM for Sequential Recommendation — Du et al. (2025) (arXiv:2507.23209, 2025)

## What this evaluates

Evaluates the ability of LLM-based sequential recommendation models to predict the next item in a user's interaction history. It specifically probes how well models capture temporal dynamics by incorporating irregular time intervals between interactions, and assesses performance under warm and cold-start conditions.

## Datasets

- **Amazon Reviews (Video Games, CDs and Vinyl, Books)** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Hit Ratio@1` **(primary)** — range: percent
  - Hit Ratio@1 measures the proportion of users for whom the ground-truth next item appears in the model's top-1 prediction. It is calculated as the number of correct top-1 predictions divided by the total number of test instances.

## Input / output format

**Input**: User interaction history sequences containing item identifiers and irregular time intervals, formatted into an optionalized text prompt for an LLM (LLaMA-2 7B).

**Output**: A single predicted item identifier/name from the candidate set.

## Scoring recipe

```python
def hit_ratio_at_1(predictions, golds):
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
    return correct / len(golds)
```

## Common pitfalls

- Using raw timestamps as text prompts instead of computed intervals, which the ablation study shows degrades performance.
- Ignoring the validation ratio metric, which tracks invalid LLM outputs (e.g., word mismatches with the candidate set), though it was 100% here due to prompt design.
- Misdefining warm/cold scenarios; they are based on the top/bottom 35% of interaction frequency or average interval length, not standard train/test splits.

## Evidence (verbatim from paper)

> Evaluation Metric. We follow previous work (Liao et al., [2024]) to use Hit Ratio@1. Notably, (Liao et al., [2024]) further adds the validation ratio as an additional metric, as certain models may generate invalid responses, such as word mismatches with the candidate set; however, in our method, the validation ratio is consistently 100% attributed to our optionalized prompt design. We adopt a leave-one-out evaluation strategy following previous work (Kang and McAuley, [2018]), where the last item in each user’s sequence is reserved for testing, the second-to-last item for validation, and the remaining items are used for training.

## Citation

```bibtex
@misc{du2025interval,
  title={Not Just What, But When: Integrating Irregular Intervals to LLM for Sequential Recommendation},
  author={Du et al. (2025)},
  year={2025},
  note={arXiv:2507.23209}
}
```

- arXiv: 2507.23209