Sql Hadoop Comparison Eval

---
name: sql-hadoop-comparison-eval
description: This evaluation compares the interactive analytics performance of four SQL-on-Hadoop systems (Impala, Drill, Spark SQL, Phoenix) by measuring query response times and resource utilization. It characterizes how each system's optimizer and execution engine handle join orders, operator selection, and data scanning across different storage formats and scaling configurations. Use when the user wants to benchmark on Unspecified SQL workloads (text/parquet), or asks about evaluating this task. Reports query_rt.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1804.00224
  bibtex_key: tapdiya2018comparative
  confidence: medium
---

# sql-hadoop-comparison-eval

> A comparative analysis of state-of-the-art SQL-on-Hadoop systems for interactive analytics — Tapdiya et al. (2018) (arXiv:1804.00224, 2018)

## What this evaluates

This evaluation compares the interactive analytics performance of four SQL-on-Hadoop systems (Impala, Drill, Spark SQL, Phoenix) by measuring query response times and resource utilization. It characterizes how each system's optimizer and execution engine handle join orders, operator selection, and data scanning across different storage formats and scaling configurations.

## Datasets

- **Unspecified SQL workloads (text/parquet)** — total ?; splits: (unstated)

## Metrics

- `query_rt` **(primary)** — range: other
  - Query Response Time (RT), measured as the total wall-clock time to execute a SQL query from submission to result retrieval.

## Input / output format

**Input**: SQL queries executed against four SQL-on-Hadoop systems (Impala, Drill, Spark SQL, Phoenix) on datasets stored in either text (row-wise) or Parquet (columnar) formats.

**Output**: Query response time (RT), resource utilization metrics, and query execution profiles containing per-operator processing times.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    system_rts = {}
    for sys, runs in predictions.items():
        rts = [run['rt'] for run in runs]
        system_rts[sys] = sum(rts) / len(rts)
    
    operator_contributions = {}
    for sys, runs in predictions.items():
        for run in runs:
            for op, time in run['operator_times'].items():
                operator_contributions.setdefault(sys, {}).setdefault(op, 0)
                operator_contributions[sys][op] += time
    return system_rts, operator_contributions
```

## Common pitfalls

- The evaluation focuses on system-level bottlenecks (CPU, Disk, Network, Memory) rather than query correctness, so exact-match validation is secondary to RT profiling.
- Scale-up and size-up experiments require strict proportional scaling of cluster nodes and data size; deviating from this breaks the comparability of linear vs. sub-linear scaling claims.

## Evidence (verbatim from paper)

> The query RT represents our performance metric. We collect resource utilization metrics and query execution profiles in each system. In each system, we aggregate the processing times for each operator type to understand the contribution of each operator type to the query RT.

## Citation

```bibtex
@misc{tapdiya2018comparative,
  title={A comparative analysis of state-of-the-art SQL-on-Hadoop systems for interactive analytics},
  author={Tapdiya et al. (2018)},
  year={2018},
  note={arXiv:1804.00224}
}
```

- arXiv: 1804.00224