R Ice Eval

---
name: r-ice-eval
description: Evaluates the accuracy of a regression framework (R-ICE) in estimating prompt-level inference carbon and energy emissions for LLMs using only token counts and publicly available performance data. It probes whether runtime can be reliably modeled as a piecewise linear function of input and output tokens without intrusive monitoring or architecture details. Use when the user wants to benchmark on HELM, or asks about evaluating this task. Reports average prediction error.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.08727
  bibtex_key: sikand2025breaking
  confidence: medium
---

# r-ice-eval

> Breaking the ICE: Exploring promises and challenges of benchmarks for Inference Carbon & Energy estimation for LLMs — Sikand et al. (2025) (arXiv:2506.08727, 2025)

## What this evaluates

Evaluates the accuracy of a regression framework (R-ICE) in estimating prompt-level inference carbon and energy emissions for LLMs using only token counts and publicly available performance data. It probes whether runtime can be reliably modeled as a piecewise linear function of input and output tokens without intrusive monitoring or architecture details.

## Datasets

- **HELM** — total ?; splits: test (-1)

## Metrics

- `average prediction error` **(primary)** — range: percent
  - Mean absolute percentage error between the estimated and actual inference carbon/energy emissions across all benchmark instances. Calculated as the average of |estimated - actual| / actual for each prompt.

## Input / output format

**Input**: Prompt text and generated output text (or token counts) from HELM benchmark instances.

**Output**: Estimated inference runtime (Idealized) and corresponding carbon/energy emissions per instance.

## Scoring recipe

```python
errors = []
for instance in dataset:
    actual = get_actual_emission(instance)
    estimated = estimate_emission(instance.prompt_tokens, instance.output_tokens)
    errors.append(abs(estimated - actual) / actual)
return mean(errors) * 100
```

## Common pitfalls

- Assumes Idealized runtime on standardized hardware, ignoring real-world variations in GPU utilization, batch sizes, and system overheads.
- Relies solely on token counts for runtime estimation, which may not capture latency spikes from KV-cache management or network I/O in distributed inference.

## Evidence (verbatim from paper)

> R-ICE introduces a benchmark-based regression framework to estimate prompt-level inference carbon and energy emissions in LLMs, leveraging publicly available performance data from HELM benchmarks. It avoids reliance on model architecture details or intrusive monitoring by modeling runtime as a function of prompt and output token counts, achieving ~15% average prediction error while enabling non-intrusive, scalable estimation for enterprise-level carbon accounting and dynamic routing decisions.

## Citation

```bibtex
@misc{sikand2025breaking,
  title={Breaking the ICE: Exploring promises and challenges of benchmarks for Inference Carbon & Energy estimation for LLMs},
  author={Sikand et al. (2025)},
  year={2025},
  note={arXiv:2506.08727}
}
```

- arXiv: 2506.08727