Evo Rnn Eval

---
name: evo-rnn-eval
description: Evaluates the ability of Evolutive RNNs (EvoRNNs) to model long-range dependencies in sequential data. It compares EvoRNNs against standard RNN baselines on language modeling and sequential recommendation tasks, emphasizing both predictive accuracy and computational efficiency. Use when the user wants to benchmark on LM1B (Billion word dataset), Sequential recommendation dataset, or asks about evaluating this task. Reports MAP@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1905.09414
  bibtex_key: [REDACTED]
  confidence: high
---

# evo-rnn-eval

> Quantifying Long Range Dependence in Language and User Behavior to improve RNNs — Belletti et al. (2019) (arXiv:1905.09414, 2019)

## What this evaluates

Evaluates the ability of Evolutive RNNs (EvoRNNs) to model long-range dependencies in sequential data. It compares EvoRNNs against standard RNN baselines on language modeling and sequential recommendation tasks, emphasizing both predictive accuracy and computational efficiency.

## Datasets

- **LM1B (Billion word dataset)** — total ?; splits: test (-1)
- **Sequential recommendation dataset** — total ?; splits: train (-1), test (-1)

## Metrics

- `MAP@20` **(primary)** — range: [0, 1]
  - Mean Average Precision at rank 20. For each user sequence, compute the average precision of the top-20 predicted items relative to the ground-truth next item. Then average this score across all sequences in the test set.

## Input / output format

**Input**: Language modeling: sequence of 128 words. Sequential recommendation: sequence of embedded item IDs representing a user's historical interactions.

**Output**: Language modeling: probability distribution over the vocabulary for the next 4 words. Sequential recommendation: predicted item ID via a softmax layer.

## Scoring recipe

```python
def compute_map_at_20(predictions, gold):
    # predictions: list of lists of top-20 predicted item IDs per query
    # gold: list of ground-truth next item IDs per query
    precisions = []
    for pred_list, true_item in zip(predictions, gold):
        hits = 0
        for rank, item in enumerate(pred_list, 1):
            if item == true_item:
                hits += 1
                precisions.append(hits / rank)
                break
        if hits == 0:
            precisions.append(0.0)
    return sum(precisions) / len(precisions)
```

## Common pitfalls

- The recommendation dataset uses a strict temporal split (last 7 days for training, last 2 days for testing). Random shuffling would cause severe data leakage.
- EvoRNNs dynamically allocate hidden units across sub-sequences, meaning computational cost (add/multiply operations) varies significantly from standard fixed-size RNNs and must be evaluated alongside predictive metrics.
- The language modeling task modifies the standard LM1B benchmark by predicting the last 4 words in a 128-word sequence, which changes the evaluation window compared to standard next-token prediction.

## Evidence (verbatim from paper)

> We report the Mean-average-precision-at-20 (MAP@20) as the main performance metric. ... The Billion word data set is a standard benchmark for language modeling aimed at predicting the next word in a text. We slightly modify the benchmark to create a long range prediction task involving longer sequences. Sequences of 128 words are considered and the model’s task is now to predict the last 4 words.

## Citation

```bibtex
@misc{belletti2019quantifying,
  title={Quantifying Long Range Dependence in Language and User Behavior to improve RNNs},
  author={Belletti et al. (2019)},
  year={2019},
  note={arXiv:1905.09414}
}
```

- arXiv: 1905.09414