Rxmrl Eval

---
name: rxmrl-eval
description: Evaluates a model's ability to maintain context and generate coherent responses in multi-turn dialogue, comparing stateful event-driven architectures against standard decoder-only LLMs. Use when the user wants to benchmark on MRL Curriculum Datasets (derived from TinyStories), or asks about evaluating this task. Reports MRL Reward Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.03561
  bibtex_key: filipek2025reactive
  confidence: high
---

# rxmrl-eval

> Reactive Transformer (RxT) -- Stateful Real-Time Processing for Event-Driven Reactive Language Models — Filipek (2025) (arXiv:2510.03561, 2025)

## What this evaluates

Evaluates a model's ability to maintain context and generate coherent responses in multi-turn dialogue, comparing stateful event-driven architectures against standard decoder-only LLMs.

## Datasets

- **MRL Curriculum Datasets (derived from TinyStories)** — total ?; splits: test (-1); repo https://github.com/RxAI-dev/rxlm

## Metrics

- `Perplexity (PPL)` — range: [0, ∞)
  - Standard cross-entropy loss exponentiated over the sequence length. Lower values indicate better fluency.
- `Accuracy` — range: [0, 100] percent
  - Percentage of correctly predicted next tokens in the sequence.
- `MRL Reward Score` **(primary)** — range: [0, 10]
  - Composite metric scaled to 0-10. Calculated as a weighted sum of BLEU score (fluency), cosine similarity to ground-truth response (immediate relevance), and cosine similarity to preceding ground-truth interaction history (long-term coherence).

## Input / output format

**Input**: Multi-turn dialogue context (up to 8+1 steps) provided as a token sequence.

**Output**: Next token or response sequence generated autoregressively.

## Scoring recipe

```python
def compute_metrics(predictions, targets, history):
    # Standard metrics
    ppl = exp(mean(cross_entropy(predictions, targets)))
    accuracy = mean(predictions == targets) * 100
    
    # MRL Reward Score (weights unspecified in text)
    bleu = compute_bleu(predictions, targets)
    cos_sim_gt = cosine_similarity(predictions, targets)
    cos_sim_hist = cosine_similarity(predictions, history)
    raw_score = 0.33 * bleu + 0.33 * cos_sim_gt + 0.33 * cos_sim_hist
    mrl_reward = scale_to_0_10(raw_score)
    
    return {'ppl': ppl, 'accuracy': accuracy, 'mrl_reward': mrl_reward}
```

## Common pitfalls

- The custom 'MRL Curriculum Datasets' are not publicly released or standardized, making independent replication difficult.
- The MRL Reward Score uses unspecified weights and relies on text embeddings for cosine similarity, which can vary significantly based on the embedding model used.
- Latency benchmarks are hardware-dependent and not normalized to FLOPs or standard inference environments.

## Evidence (verbatim from paper)

> Performance was assessed using three metrics: Perplexity (PPL): A standard measure of a language model’s ability to predict a sequence of text. Lower values indicate better fluency. Accuracy: Standard next-token prediction accuracy. MRL Reward Score: A custom, composite metric designed to serve as a proxy for conversational quality, scaled to a 0-10 range. It is a weighted sum of BLEU score (for fluency), cosine similarity between the generated response and the ground-truth response (for immediate relevance), and previous interaction content, with cosine similarity between the generated response and the history of preceding ground-truth interactions (for long-term coherence).

## Citation

```bibtex
@misc{filipek2025reactive,
  title={Reactive Transformer (RxT) -- Stateful Real-Time Processing for Event-Driven Reactive Language Models},
  author={Filipek (2025)},
  year={2025},
  note={arXiv:2510.03561}
}
```

- arXiv: 2510.03561