Next Transaction Prediction Eval

---
name: next-transaction-prediction-eval
description: Evaluates a model's ability to predict the next user transaction or item interaction based on historical sequential behavior. It probes the model's capacity to capture periodic patterns, user-specific preferences, and generalize across financial and recommendation domains. Use when the user wants to benchmark on WeChat Pay, CCT, MBD-mini, MovieLens-1M, Yelp, or asks about evaluating this task. Reports HR@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.10102
  bibtex_key: li2025panther
  confidence: high
---

# next-transaction-prediction-eval

> PANTHER: Generative Pretraining Beyond Language for Sequential User Behavior Modeling — Li et al. (2025) (arXiv:2510.10102, 2025)

## What this evaluates

Evaluates a model's ability to predict the next user transaction or item interaction based on historical sequential behavior. It probes the model's capacity to capture periodic patterns, user-specific preferences, and generalize across financial and recommendation domains.

## Datasets

- **WeChat Pay** — total 5300000000; splits: train (-1), val (-1), test (-1); repo https://github.com/yzhangjy/PANTHER
- **CCT** — total 20000000; splits: train (-1), val (-1), test (-1)
- **MBD-mini** — total ?; splits: train (-1), val (-1), test (-1)
- **MovieLens-1M** — total 1000000; splits: train (-1), val (-1), test (-1); HF `ml-1m`
- **Yelp** — total ?; splits: train (-1), val (-1), test (-1); HF `yelp`

## Metrics

- `HR@1` **(primary)** — range: [0, 1]
  - Hit Ratio at 1: 1 if the ground-truth item is ranked #1 in the predicted list, else 0. Averaged over test instances.
- `HR@5` — range: [0, 1]
  - Hit Ratio at 5: 1 if the ground-truth item appears in the top-5 predicted items, else 0. Averaged over test instances.
- `HR@10` — range: [0, 1]
  - Hit Ratio at 10: 1 if the ground-truth item appears in the top-10 predicted items, else 0. Averaged over test instances.
- `NDCG@5` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at 5: Ranks predicted items by score, discounts gains logarithmically by position, and normalizes by the ideal DCG. Averaged over test instances.

## Input / output format

**Input**: Tokenized sequence of historical user transactions (attributes: amount, merchant category, etc.)

**Output**: Ranked list of candidate next transactions/items, or top-K predictions.

## Scoring recipe

```python
def compute_hr_at_k(preds, gold, k):
    return 1.0 if gold in preds[:k] else 0.0

def compute_ndcg_at_k(preds, gold, k):
    dcg = 0.0
    for i, item in enumerate(preds[:k]):
        if item == gold:
            dcg = 1.0 / math.log2(i + 2)
            break
    idcg = 1.0 / math.log2(2)
    return dcg / idcg if idcg > 0 else 0.0

# Aggregate over test set
hr_scores = [compute_hr_at_k(preds, gold, k) for preds, gold in test_data]
ndcg_scores = [compute_ndcg_at_k(preds, gold, k) for preds, gold in test_data]
return sum(hr_scores) / len(hr_scores), sum(ndcg_scores) / len(ndcg_scores)
```

## Common pitfalls

- Must use full-ranking evaluation (~3,700 items) rather than sampled negatives to match the paper's protocol.
- Splits are chronological, not random, to prevent data leakage from future transactions.

## Evidence (verbatim from paper)

> Specifically, HR@K measures the fraction of test instances in which the ground-truth item appears among the top-K predicted items. NDCG@K assesses the ranking quality by assigning higher weights to relevant items placed at top positions, normalized by the ideal discounted gain.

## Citation

```bibtex
@misc{li2025panther,
  title={PANTHER: Generative Pretraining Beyond Language for Sequential User Behavior Modeling},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2510.10102}
}
```

- arXiv: 2510.10102