Tfrb Eval

---
name: tfrb-eval
description: Evaluates the causal reasoning and forecasting accuracy of LLMs and time-series models on multi-domain time-series data. It probes whether step-by-step reasoning and external event context improve numerical predictions or introduce narrative bias, particularly in stochastic versus pattern-rich domains. Use when the user wants to benchmark on TFRBench, or asks about evaluating this task. Reports MASE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.05364
  bibtex_key: ahamed2026tfrbench
  confidence: high
---

# tfrb-eval

> TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems — Ahamed et al. (2026) (arXiv:2604.05364, 2026)

## What this evaluates

Evaluates the causal reasoning and forecasting accuracy of LLMs and time-series models on multi-domain time-series data. It probes whether step-by-step reasoning and external event context improve numerical predictions or introduce narrative bias, particularly in stochastic versus pattern-rich domains.

## Datasets

- **TFRBench** — total ?; splits: test (-1)

## Metrics

- `MASE` **(primary)** — range: other
  - Mean Absolute Scaled Error: MASE = MAE_model / MAE_naive, where the naive forecast uses the previous time step. Lower values indicate better forecasting accuracy.
- `LLM-as-Judge Score` — range: [1, 5]
  - LLM-evaluated reasoning quality scored on a 1-5 Likert scale across four dimensions: domain relevance, forecasting correctness, event relevance, and logic consistency. The average score across dimensions and samples is reported.
- `Overall Success Rate` — range: percent
  - Percentage of test samples where the model's MASE is strictly less than 1.0.

## Input / output format

**Input**: Historical time-series context window (potentially augmented with external event/holiday data). Models receive system prompts enforcing role adherence and are asked to either output a direct numerical forecast or generate a step-by-step reasoning trace before forecasting.

**Output**: Either a single numerical prediction, or a structured reasoning trace (analyzing trends, seasonality, cross-channel dependencies, and event predictions) followed by a final numerical forecast.

## Scoring recipe

```python
def compute_mase(predictions, gold):
    mae_model = mean(abs(predictions - gold))
    mae_naive = mean(abs(gold[1:] - gold[:-1]))
    return mae_model / mae_naive

def compute_success_rate(predictions, gold):
    mases = [compute_mase(p, g) for p, g in zip(predictions, gold)]
    return sum(1 for m in mases if m < 1.0) / len(mases) * 100

def compute_judge_score(reasoning_traces):
    # Gemini-3-Pro evaluates each trace on 4 dimensions (1-5 scale)
    # Returns average score across dimensions and samples
    return llm_judge.evaluate(traces, dimensions=[
        'domain_relevance', 'forecasting_correctness',
        'event_relevance', 'logic_consistency'
    ])
```

## Common pitfalls

- In stochastic domains like Finance, step-by-step reasoning can introduce 'narrative bias' where models hallucinate causal links, actually degrading forecasting accuracy compared to direct prediction.
- Event forecasting with external search acts as a distractor in high-entropy technical workloads (e.g., Web/CloudOps), causing significant error spikes when models over-rely on irrelevant external news.
- Evaluating only final numerical accuracy without auditing reasoning traces misses critical failures in causal attribution and logic consistency.

## Evidence (verbatim from paper)

> Figure 4 reports MASE across three distinct experimental settings for the corresponding models. We report the mean and standard deviation $(\mathrm{mean}_{\mathrm{std}})$ across three runs for statistical significance. Default parameters are used for reasoning models, while the "w/ Reasoning" setting enforces an external CoT trace. Detailed results are provided in App. J.

## Citation

```bibtex
@misc{ahamed2026tfrbench,
  title={TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems},
  author={Ahamed et al. (2026)},
  year={2026},
  note={arXiv:2604.05364}
}
```

- arXiv: 2604.05364