Siwarex Eval

---
name: siwarex-eval
description: Evaluates LLM-based natural language question answering over heterogeneous data sources by testing the model's ability to generate SQL queries that correctly invoke both database tables and external APIs. It probes complex query planning, API sequencing, and routing across mixed data modalities. Use when the user wants to benchmark on Spider (modified with API-replaced tables), or asks about evaluating this task. Reports execution accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.05735
  bibtex_key: fokoue2024siwarex
  confidence: high
---

# siwarex-eval

> A System and Benchmark for LLM-based Q&A on Heterogeneous Data — Fokoue et al. (2024) (arXiv:2409.05735, 2024)

## What this evaluates

Evaluates LLM-based natural language question answering over heterogeneous data sources by testing the model's ability to generate SQL queries that correctly invoke both database tables and external APIs. It probes complex query planning, API sequencing, and routing across mixed data modalities.

## Datasets

- **Spider (modified with API-replaced tables)** — total ?; splits: test (-1)

## Metrics

- `execution accuracy` **(primary)** — range: [0, 1]
  - Execution accuracy measures the fraction of questions where the model's generated query produces results identical to the gold standard SQL query's results on the database. Uses the sophisticated matching approach from Zhong et al. (2020) to handle result set equivalence.

## Input / output format

**Input**: Natural language question

**Output**: SQL query (potentially containing User-Defined Function calls for API invocations)

## Scoring recipe

```python
def compute_execution_accuracy(predictions, gold_queries, db):
    correct = 0
    for pred_sql, gold_sql in zip(predictions, gold_queries):
        pred_res = execute_sql(pred_sql, db)
        gold_res = execute_sql(gold_sql, db)
        if sophisticated_match(pred_res, gold_res): # Zhong et al. 2020
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Models often fail at sequencing multiple API calls or merging/aggregating their results.
- Routing errors occur when the LLM decomposes a question correctly but sends it to the wrong tool (DB vs API).
- API argument hallucination leads to incorrect function invocations even when the right API is selected.

## Evidence (verbatim from paper)

> The evaluation metric is the execution accuracy measured by comparing the results produced by our system (or the baseline system) against those produced by the evaluation of the gold standard spider sql query on the original spider db. We use the sophisticated comparison approach that was introduced by (Zhong et al., 2020).

## Citation

```bibtex
@misc{fokoue2024siwarex,
  title={A System and Benchmark for LLM-based Q&A on Heterogeneous Data},
  author={Fokoue et al. (2024)},
  year={2024},
  note={arXiv:2409.05735}
}
```

- arXiv: 2409.05735