Salesforce Bi Eval

---
name: salesforce_bi_eval
description: Evaluates the quality of synthetically generated Text-to-SQL data by measuring question-SQL alignment and business realism in a sales analytics domain. Use when the user wants to benchmark on Salesforce Sales Analytics Database, or asks about evaluating this task. Reports Question-SQL Alignment (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.14518
  bibtex_key: [REDACTED]
  confidence: medium
---

# salesforce_bi_eval

> Business Logic-Driven Text-to-SQL Data Synthesis for Business Intelligence — Liu et al. (2026) (arXiv:2601.14518, 2026)

## What this evaluates

Evaluates the quality of synthetically generated Text-to-SQL data by measuring question-SQL alignment and business realism in a sales analytics domain.

## Datasets

- **Salesforce Sales Analytics Database** — total 240; splits: test (240)

## Metrics

- `Question-SQL Alignment (%)` **(primary)** — range: percent
  - Percentage of generated question-SQL pairs where the SQL query correctly answers the natural language question, validated against the target schema.
- `Question Realism (%)` — range: percent
  - Percentage of generated questions rated as realistic and aligned with actual business intelligence workflows and personas.

## Input / output format

**Input**: Business logic instances (persona, work scenario, workflow) and target database schema (SOQL).

**Output**: Natural language questions paired with corresponding SOQL queries.

## Scoring recipe

```python
def calculate_metrics(pairs):
    aligned = 0
    realistic = 0
    for q, sql in pairs:
        if verify_alignment(q, sql, schema): aligned += 1
        if verify_realism(q, domain_experts): realistic += 1
    return aligned / len(pairs), realistic / len(pairs)
```

## Common pitfalls

- The database uses SOQL, not standard SQL, which affects query execution and compatibility with standard Text-to-SQL benchmarks.
- Data quality metrics rely on expert validation or proprietary scoring rubrics not fully detailed in the text.
- The dataset is private and production-scale, limiting direct reproducibility without access to the Salesforce schema.

## Evidence (verbatim from paper)

> In total, we generate 240 question–SQL pairs, spanning diverse personas and business reasoning complexity levels. Table 2: Data quality comparison with existing Text-to-SQL data generation methods. MethodEvaluation LLM Question-SQLAlignment(%) QuestionRealism(%) OmniSQL*Li et al. ([2025a])* Gemini3 Pro95.47 $(\downarrow 3.12)$78.91 $(\downarrow 19.53)$

## Citation

```bibtex
@misc{liu2026businesslogic,
  title={Business Logic-Driven Text-to-SQL Data Synthesis for Business Intelligence},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2601.14518}
}
```

- arXiv: 2601.14518