Od Llm Eval

---
name: od-llm-eval
description: Evaluates the ability of compressed large language models to perform sequential recommendation on resource-constrained devices. It probes how well a model preserves ranking quality and recommendation accuracy after aggressive parameter compression (SVD + normalization) while maintaining low latency. Use when the user wants to benchmark on Amazon Instruments, Games, Arts, or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.09306
  bibtex_key: xia2026ondevice
  confidence: high
---

# od-llm-eval

> On-Device Large Language Models for Sequential Recommendation — Xin Xia et al. (2026) (arXiv:2601.09306, 2026)

## What this evaluates

Evaluates the ability of compressed large language models to perform sequential recommendation on resource-constrained devices. It probes how well a model preserves ranking quality and recommendation accuracy after aggressive parameter compression (SVD + normalization) while maintaining low latency.

## Datasets

- **Amazon Instruments, Games, Arts** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `HR@5` — range: [0, 1]
  - Hit Ratio at cutoff K: fraction of test users for whom the true next item appears in the top-K predicted list.
- `HR@10` — range: [0, 1]
  - Hit Ratio at cutoff K: fraction of test users for whom the true next item appears in the top-K predicted list.
- `NDCG@5` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff K: weighted sum of relevance scores (1 for hit) discounted by rank position, normalized by the ideal DCG.
- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff K: weighted sum of relevance scores (1 for hit) discounted by rank position, normalized by the ideal DCG.

## Input / output format

**Input**: User interaction history sequences represented as item titles and descriptions, formatted as prompts for the LLM.

**Output**: A ranked list of top-K candidate items (K=5 or 10) generated by the model.

## Scoring recipe

```python
def evaluate(gold_item, pred_list, k):
    top_k = pred_list[:k]
    hr = 1.0 if gold_item in top_k else 0.0
    dcg = sum(1.0 / math.log2(i + 2) for i, item in enumerate(top_k) if item == gold_item)
    idcg = 1.0 / math.log2(2)
    ndcg = dcg / idcg
    return hr, ndcg
```

## Common pitfalls

- The validation and test sets are constructed deterministically using the last two interactions per user sequence, not via random or temporal splits.
- Main results assume a fixed 50% compression ratio; varying this ratio significantly impacts HR/NDCG scores.
- Calibration set size (256 samples) is fixed for the primary evaluation; changing it alters the one-shot SVD decomposition quality.

## Evidence (verbatim from paper)

> For any sequence $S_{u}\={v_{1}^{u},v_{2}^{u},\ldots,v_{\ell}^{u}}$, the last two interacting items, $v_{\ell}^{u},v_{\ell-1}^{u}$, are used as the validation and test set respectively. Since our method is designed for sequential recommendation tasks, we use Hit Ratio and NDCG (Normalized Discounted Cumulative Gain) to evaluate the recommendation results of the top-5 and top-10 results.

## Citation

```bibtex
@misc{xia2026ondevice,
  title={On-Device Large Language Models for Sequential Recommendation},
  author={Xin Xia et al. (2026)},
  year={2026},
  note={arXiv:2601.09306}
}
```

- arXiv: 2601.09306