Hiespec Throughput Eval

---
name: hiespec-throughput-eval
description: Evaluates the inference throughput speedup of hierarchical speculative decoding against vanilla auto-regressive decoding and other acceleration baselines. It probes the method's ability to accelerate token generation across dialogue, summarization, code generation, and mathematical reasoning tasks without relying on auxiliary draft models. Use when the user wants to benchmark on ShareGPT, CNN/DM, XSum, HumanEval, GSM8K, or asks about evaluating this task. Reports Speedup (vs. Vanilla).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.01336
  bibtex_key: kumar2025hiespec
  confidence: high
---

# hiespec-throughput-eval

> HiSpec: Hierarchical Speculative Decoding for LLMs — Kumar et al. (2025) (arXiv:2510.01336, 2025)

## What this evaluates

Evaluates the inference throughput speedup of hierarchical speculative decoding against vanilla auto-regressive decoding and other acceleration baselines. It probes the method's ability to accelerate token generation across dialogue, summarization, code generation, and mathematical reasoning tasks without relying on auxiliary draft models.

## Datasets

- **ShareGPT** — total ?; splits: test (-1)
- **CNN/DM** — total ?; splits: test (-1)
- **XSum** — total ?; splits: test (-1)
- **HumanEval** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)

## Metrics

- `Speedup (vs. Vanilla)` **(primary)** — range: ratio
  - Ratio of the evaluated method's throughput to the throughput of vanilla auto-regressive decoding on the same hardware and configuration. Higher values indicate greater acceleration.

## Input / output format

**Input**: Natural language prompts for dialogue continuation, text summarization, code completion, and mathematical reasoning problems.

**Output**: Autoregressively generated token sequences until an end-of-sequence token or maximum length is reached.

## Scoring recipe

```python
def compute_speedup(method_throughput, vanilla_throughput):
    return method_throughput / vanilla_throughput
# Evaluated per dataset and model size combination
```

## Common pitfalls

- Speedup is a relative ratio to vanilla decoding, not absolute tokens/sec, making cross-hardware comparisons invalid.
- The table only reports throughput; accuracy preservation is claimed in text but not quantified in this specific evaluation table.
- Comparisons across different base model sizes (e.g., 7B vs 70B) are presented in separate blocks, which can obscure per-model scaling trends.

## Evidence (verbatim from paper)

> HiSpec improves throughput: Table[1] compares the performance of HiSpec against prior works which accelerate draft token generation. HiSpec improves throughput by $1.7	imes$ on average and up to $2.08	imes$ compared to vanilla auto-regressive decoding. *Table 1: Throughput of HiSpec and other competing approaches relative to vanilla auto-regressive decoding (higher is better), across dialogue (ShareGPT), text summarization (CNN/DM, XSum), code generation (HumanEval), and mathematical reasoning (GSM8K) tasks.* | Method | Speedup (vs. Vanilla) |

## Citation

```bibtex
@misc{kumar2025hiespec,
  title={HiSpec: Hierarchical Speculative Decoding for LLMs},
  author={Kumar et al. (2025)},
  year={2025},
  note={arXiv:2510.01336}
}
```

- arXiv: 2510.01336