Loopserv Eval

---
name: loopserv-eval
description: Evaluates an adaptive dual-phase LLM inference acceleration system for multi-turn dialogues, probing its ability to maintain generation accuracy and reduce computational overhead across varying query positions in long-context conversations. It specifically tests whether the system can generalize beyond positional heuristics used by static KV cache compression methods. Use when the user wants to benchmark on MFQA-en, 2WikiMQA, Musique, HotpotQA, NrtvQA, Qasper, MultiNews, GovReport, QMSum, TREC, SAMSUM, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.13681
  bibtex_key: li2025loopserv
  confidence: high
---

# loopserv-eval

> LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialogues — Li et al. (2025) (arXiv:2507.13681, 2025)

## What this evaluates

Evaluates an adaptive dual-phase LLM inference acceleration system for multi-turn dialogues, probing its ability to maintain generation accuracy and reduce computational overhead across varying query positions in long-context conversations. It specifically tests whether the system can generalize beyond positional heuristics used by static KV cache compression methods.

## Datasets

- **MFQA-en** — total ?; splits: test (-1)
- **2WikiMQA** — total ?; splits: test (-1)
- **Musique** — total ?; splits: test (-1)
- **HotpotQA** — total ?; splits: test (-1)
- **NrtvQA** — total ?; splits: test (-1)
- **Qasper** — total ?; splits: test (-1)
- **MultiNews** — total ?; splits: test (-1)
- **GovReport** — total ?; splits: test (-1)
- **QMSum** — total ?; splits: test (-1)
- **TREC** — total ?; splits: test (-1)
- **SAMSUM** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Exact match ratio: fraction of predictions that exactly match the ground truth label or answer. Standard for QA and Few-shot classification tasks.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall, typically computed as macro or micro average depending on the dataset. Used for QA tasks where partial matches or entity overlap matter.
- `Rouge-L` — range: [0, 1]
  - Recall-Oriented Understudy for Gisting Evaluation (Longest Common Subsequence). Measures the longest matching subsequence between generated summary and reference summary, normalized by reference length.

## Input / output format

**Input**: Multi-turn dialogue context containing multiple rounds with diverse query positions (beginning, middle, or end) and dependencies. Covers Question Answering, Summarization, and Few-shot Learning tasks.

**Output**: Model-generated text response or class label corresponding to the target query in the dialogue turn.

## Scoring recipe

```python
def evaluate(predictions, golds, task_type):
    if task_type == 'QA':
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    elif task_type == 'Summarization':
        return rouge_l_score(predictions, golds)
    elif task_type == 'FewShot':
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    return 0.0
```

## Common pitfalls

- Assuming KV cache baselines generalize well across all query positions; they often degrade significantly when queries are at the beginning or middle due to reliance on positional heuristics.
- Setting the token budget B or sparsity threshold α too aggressively without checking task-specific sensitivity, which can cause disproportionate accuracy drops without meaningful efficiency gains.

## Evidence (verbatim from paper)

> For each dataset, we compare LoopServe with six state-of-the-art KV cache acceleration baselines and two base LLMs, using F1, Rouge-L, or Accuracy as appropriate. As shown in Table 1, LoopServe achieves the best or comparable results across most datasets and query positions.

## Citation

```bibtex
@misc{li2025loopserv,
  title={LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialogues},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2507.13681}
}
```

- arXiv: 2507.13681