Indicdb Eval

---
name: indicdb-eval
description: Evaluates cross-lingual semantic parsing and Text-to-SQL capabilities across Indian languages, probing a model's ability to link natural language queries to complex, high-join-depth relational schemas and generate syntactically and semantically correct SQL. Use when the user wants to benchmark on IndicDB, or asks about evaluating this task. Reports Execution Accuracy (EX).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.13686
  bibtex_key: [REDACTED]
  confidence: medium
---

# indicdb-eval

> IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages — Dawar et al. (2026) (arXiv:2604.13686, 2026)

## What this evaluates

Evaluates cross-lingual semantic parsing and Text-to-SQL capabilities across Indian languages, probing a model's ability to link natural language queries to complex, high-join-depth relational schemas and generate syntactically and semantically correct SQL.

## Datasets

- **IndicDB** — total 15617; splits: test (15617)

## Metrics

- `Execution Accuracy (EX)` **(primary)** — range: [0, 1]
  - Standard Text-to-SQL metric: the predicted SQL is executed on the target database, and its result set is compared against the result set of the gold SQL. A match yields a score of 1, otherwise 0. Averaged over all instances.

## Input / output format

**Input**: Natural language question (in English or one of six Indic languages) and the target database schema (table names, column names, foreign key relationships, and sample values).

**Output**: A single SQL query string.

## Scoring recipe

```python
def evaluate(predictions, golds, databases):
    correct = 0
    for pred_sql, gold_sql, db in zip(predictions, golds, databases):
        pred_res = execute_sql(db, pred_sql)
        gold_res = execute_sql(db, gold_sql)
        if sets_equal(pred_res, gold_res):
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Models often hallucinate filter values not present in the database; the benchmark mitigates this by grounding SQL generation in actual database values.
- Translation errors in Indic languages can invert sorting directions or swap entities, leading to logically valid but semantically wrong SQL; rigorous HITL verification is required to isolate model reasoning gaps from translation artifacts.
- Schema-linking requires navigating deep foreign-key chains; models may fail to identify the minimal relevant sub-schema or misinterpret join paths.

## Evidence (verbatim from paper)

> The English dataset contains 3,684 validated natural language query and SQL pairs with a calibrated difficulty distribution as shown in Table 1. Logical and syntactic integrity is maintained through a two-tier validation protocol involving PostgreSQL execution and a manual audit by three database experts. The semantic alignment between Indic queries and SQL logic was confirmed using the Fleiss’ Kappa (κ) statistic, which yielded a coefficient of 0.84.

## Citation

```bibtex
@misc{dawar2026indicdb,
  title={IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages},
  author={Dawar et al. (2026)},
  year={2026},
  note={arXiv:2604.13686}
}
```

- arXiv: 2604.13686