Compound Ai Hw Sw Bench Eval

---
name: compound-ai-hw-sw-bench-eval
description: This benchmark evaluates hardware-software co-design trade-offs for compound AI applications by measuring end-to-end latency, energy consumption, and accuracy across multi-modal workflows like video QA, evolutionary code generation, and RAG. It probes how different hardware configurations and software optimizations impact system performance under varying latency targets and workload patterns. Use when the user wants to benchmark on Google FRAMES benchmark, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.09593
  bibtex_key: [REDACTED]
  confidence: medium
---

# compound-ai-hw-sw-bench-eval

> Benchmarking Compound AI Applications for Hardware-Software Co-Design — Samuthrsindh et al. (2026) (arXiv:2604.09593, 2026)

## What this evaluates

This benchmark evaluates hardware-software co-design trade-offs for compound AI applications by measuring end-to-end latency, energy consumption, and accuracy across multi-modal workflows like video QA, evolutionary code generation, and RAG. It probes how different hardware configurations and software optimizations impact system performance under varying latency targets and workload patterns.

## Datasets

- **Google FRAMES benchmark** — total 824; splits: test (824)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard exact-match or semantic similarity score comparing the generated answer to the ground truth reference.
- `latency` — range: seconds
  - Total time from query submission to final response generation, measured in seconds or milliseconds.
- `energy consumption` — range: joules
  - Total power draw integrated over the execution time of the workflow, measured in joules.

## Input / output format

**Input**: User queries (text, video frames, audio) submitted to the compound AI pipeline, which processes them through modality-specific encoders, LLMs, and vector databases.

**Output**: Final generated answers or optimized code, along with system-level telemetry metrics (latency, energy, CPU/GPU utilization, cache hit rates) collected via vLLM, SAR, and DCGMI.

## Scoring recipe

```python
def evaluate_workflow(workflow, queries, hardware_config):
    total_latency = 0
    total_energy = 0
    correct_answers = 0
    for q in queries:
        start_time = time.now()
        start_power = measure_power()
        response = workflow.run(q, hardware_config)
        end_time = time.now()
        end_power = measure_power()
        latency = end_time - start_time
        energy = integrate_power(start_power, end_power, latency)
        total_latency += latency
        total_energy += energy
        if check_accuracy(response, q.gold):
            correct_answers += 1
    return {
        'avg_latency': total_latency / len(queries),
        'total_energy': total_energy,
        'accuracy': correct_answers / len(queries)
    }
```

## Common pitfalls

- System-level metrics like latency and energy are highly sensitive to background processes and hardware state; the paper notes the need for controlled load generation (Poisson distribution) to simulate realistic usage.
- Accuracy evaluation depends heavily on the embedding model and chunking strategy (e.g., 2,000-token chunks with 200-token overlap), which can significantly alter retrieval quality and downstream LLM performance.

## Evidence (verbatim from paper)

> For the accuracy–latency tradeoff experiment, documents were embedded using a Text-Embedding-3-Small and split into 2,000-token chunks with a 200-token overlap. The system retrieved the top $k$ chunks, with $k$ ranging from 5 to 30, and was evaluated on the Google FRAMES benchmark across three database configurations: a baseline, a 100-question, and the full 824-question database.

## Citation

```bibtex
@misc{samuthrsindh2026compoundai,
  title={Benchmarking Compound AI Applications for Hardware-Software Co-Design},
  author={Samuthrsindh et al. (2026)},
  year={2026},
  note={arXiv:2604.09593}
}
```

- arXiv: 2604.09593