Tenspiler Eval

---
name: tenspiler-eval
description: Evaluates the correctness and performance of a verified lifting-based compiler that transpiles sequential C++/Python code into tensor operations across various DSLs and hardware accelerators. It measures synthesis efficiency, kernel execution speedup, and end-to-end performance including data transfer overhead. Use when the user wants to benchmark on TENSPILER benchmark suite, or asks about evaluating this task. Reports kernel_performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.18249
  bibtex_key: qiu2024tenspiler
  confidence: high
---

# tenspiler-eval

> Tenspiler: A Verified Lifting-Based Compiler for Tensor Operations (Extended Version) — Qiu et al. (2024) (arXiv:2404.18249, 2024)

## What this evaluates

Evaluates the correctness and performance of a verified lifting-based compiler that transpiles sequential C++/Python code into tensor operations across various DSLs and hardware accelerators. It measures synthesis efficiency, kernel execution speedup, and end-to-end performance including data transfer overhead.

## Datasets

- **TENSPILER benchmark suite** — total 69; splits: test (69)

## Metrics

- `kernel_performance` **(primary)** — range: other
  - Ratio of baseline C++ execution time to translated code execution time, excluding data transfer overhead.
- `end_to_end_performance` — range: other
  - Ratio of baseline C++ execution time to translated code execution time, including initial setup and data movement overhead.
- `synthesis_time` — range: seconds
  - Wall-clock time taken by the synthesis and verification algorithm to generate a correct translation, capped at a 1-hour timeout.

## Input / output format

**Input**: Sequential C++ or Python functions (image processing, DL, linear algebra, signal processing kernels) paired with input data (grayscale ImageNet images, Vicuna model weights, or random float vectors).

**Output**: Translated tensor operation code in a target DSL (NumPy, TensorFlow, PyTorch, MLX, TPC-C, or Gemmini) and its measured execution time on specified hardware.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    baseline_time = run_baseline(gold)
    translated_code = predictions['translated_code']
    translated_time = run_translated(translated_code)
    kernel_perf = baseline_time / translated_time
    e2e_time = translated_time + data_transfer_overhead
    e2e_perf = baseline_time / e2e_time
    return kernel_perf, e2e_perf
```

## Common pitfalls

- Kernel performance excludes data transfer overhead, while end-to-end performance includes it; confusing the two leads to incorrect performance claims.
- Gemmini accelerator requires square matrix inputs, necessitating padding that inflates data volume and skews performance if not accounted for.
- Synthesis timeout is 1 hour; benchmarks failing to synthesize within this limit are excluded from average timing calculations.

## Evidence (verbatim from paper)

> Kernel performance focuses on computation time excluding data transfer overhead. We see significant improvements as illustrated in Figures 9, with an average speedup of 105.1× across all benchmarks. ... End-to-end Performance. While frameworks and accelerators deliver substantial kernel performance enhancements, a comprehensive assessment must account for end-to-end benchmark times, encompassing initial setup and data movement between the host (CPU) and the accelerator device.

## Citation

```bibtex
@misc{qiu2024tenspiler,
  title={Tenspiler: A Verified Lifting-Based Compiler for Tensor Operations (Extended Version)},
  author={Qiu et al. (2024)},
  year={2024},
  note={arXiv:2404.18249}
}
```

- arXiv: 2404.18249