Tqabench Eval

---
name: tqabench-eval
description: Evaluates large language models' ability to perform multi-table question answering across varying context lengths (8K–64K tokens) and complex reasoning tasks. It probes cross-table inference, symbolic reasoning, and handling of real-world relational data without Wikipedia bias. Use when the user wants to benchmark on TQA-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.19504
  bibtex_key: qiu2024tqabench
  confidence: medium
---

# tqabench-eval

> TQA-Bench: Evaluating LLMs for Multi-Table Question Answering with Scalable Context and Symbolic Extension — Qiu et al. (2024) (arXiv:2411.19504, 2024)

## What this evaluates

Evaluates large language models' ability to perform multi-table question answering across varying context lengths (8K–64K tokens) and complex reasoning tasks. It probes cross-table inference, symbolic reasoning, and handling of real-world relational data without Wikipedia bias.

## Datasets

- **TQA-Bench** — total ?; splits: test (-1); repo https://github.com/Relaxed-System-Lab/TQA-Bench

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact-match accuracy computed by comparing the model's predicted answer string against the ground truth answer generated via Python scripts. A prediction is correct if it exactly matches the gold answer after standard normalization.

## Input / output format

**Input**: Serialized multi-table database in Markdown format (including table names and foreign key relationships) concatenated with a natural language question. Context length varies from 8K to 64K tokens.

**Output**: Natural language answer string corresponding to the question.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_text(pred) == normalize_text(gold):
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Models often fail when serialized tables exceed 8K tokens due to context window limits, despite the benchmark's scalable sampling design.
- Naive row sampling breaks foreign key referential integrity, invalidating cross-table queries and symbolic extensions.
- Symbolic augmentations require multi-step logical reasoning rather than simple pattern matching, causing significant performance drops on complex calculation tasks.

## Evidence (verbatim from paper)

> For each question category, we develop structured question templates that are augmented with symbolic extensions to assess reasoning capabilities beyond simple retrieval. These templates are paired with Python-based answer generation, enabling the automated creation of benchmark questions and ensuring scalability and reliability in task evaluation.

## Citation

```bibtex
@misc{qiu2024tqabench,
  title={TQA-Bench: Evaluating LLMs for Multi-Table Question Answering with Scalable Context and Symbolic Extension},
  author={Qiu et al. (2024)},
  year={2024},
  note={arXiv:2411.19504}
}
```

- arXiv: 2411.19504