Model Selection Energy Eval

---
name: model-selection-energy-eval
description: This evaluation protocol assesses the trade-off between model size (parameter count) and task utility across multiple AI benchmarks. It aims to identify energy-efficient models that maintain high performance, enabling estimation of global AI inference energy savings through strategic model selection. Use when the user wants to benchmark on OpenLLM Leaderboard, LMSys Chatbot Arena, NPHardEval, BigCode Leaderboard, mtebLeaderboard, WMT English-German, Open Object Detection Leaderboard, ImageNet, Semantic Segmentation on ADE20K, Open ASR Leaderboard, ARCH, GenAI, MMMU Benchmark, Eth1-336, or asks about evaluating this task. Reports Utility.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.01889
  bibtex_key: [REDACTED]
  confidence: medium
---

# model-selection-energy-eval

> Small is Sufficient: Reducing the World AI Energy Consumption Through Model Selection — da Silva Barros et al. (2025) (arXiv:2510.01889, 2025)

## What this evaluates

This evaluation protocol assesses the trade-off between model size (parameter count) and task utility across multiple AI benchmarks. It aims to identify energy-efficient models that maintain high performance, enabling estimation of global AI inference energy savings through strategic model selection.

## Datasets

- **OpenLLM Leaderboard** — total ?; splits: (unstated)
- **LMSys Chatbot Arena** — total ?; splits: (unstated)
- **NPHardEval** — total ?; splits: (unstated)
- **BigCode Leaderboard** — total ?; splits: (unstated)
- **mtebLeaderboard** — total ?; splits: (unstated)
- **WMT English-German** — total ?; splits: (unstated)
- **Open Object Detection Leaderboard** — total ?; splits: (unstated)
- **ImageNet** — total ?; splits: (unstated)
- **Semantic Segmentation on ADE20K** — total ?; splits: (unstated)
- **Open ASR Leaderboard** — total ?; splits: (unstated)
- **ARCH** — total ?; splits: (unstated)
- **GenAI** — total ?; splits: (unstated)
- **MMMU Benchmark** — total ?; splits: (unstated)
- **Eth1-336** — total ?; splits: (unstated)

## Metrics

- `Utility` **(primary)** — range: [0, 100] or benchmark-specific
  - Normalized performance score reported by the respective benchmark or leaderboard, used to quantify model capability for a given task.
- `Energy(J)` — range: other
  - Estimated inference energy consumption in Joules per model-task instance, derived from empirical latency, throughput, or hardware power draw metrics.

## Input / output format

**Input**: Model parameter count, benchmark performance scores, and hardware/inference specifications for each model-task pair.

**Output**: Utility value, parameter count, and estimated inference energy (Joules) per model, along with selection of 'efficient' vs 'best-performing' models.

## Scoring recipe

```python
# 1. Aggregate model performance (Utility) and parameter counts from public leaderboards
# 2. Estimate inference energy (Energy(J)) using empirical latency/throughput or hardware power draw
# 3. Plot Utility vs. parameters to identify Pareto frontier
# 4. Select 'efficient' model: highest Utility-to-size ratio on frontier
# 5. Select 'best' model: absolute maximum Utility
# 6. Compute savings: (Energy_best - Energy_efficient) / Energy_best
```

## Common pitfalls

- Utility scores are benchmark-specific and not directly comparable across different AI tasks.
- Energy estimates rely on hardware assumptions and batch sizes that vary across leaderboards.
- Model adoption is proxied by Hugging Face download counts, which may not reflect actual inference volume.

## Evidence (verbatim from paper)

> | Task | Model | Params | Utility | Energy(J) | Downloads |
| --- | --- | --- | --- | --- | --- |
| Text Generation | internlm/internlm2_5-7b-chat (efficient) | 8B | 0.6 | 8035.0 | 37281 |

## Citation

```bibtex
@misc{dasilvabarros2025small,
  title={Small is Sufficient: Reducing the World AI Energy Consumption Through Model Selection},
  author={da Silva Barros et al. (2025)},
  year={2025},
  note={arXiv:2510.01889}
}
```

- arXiv: 2510.01889