Data Product Discovery Eval

---
name: data-product-discovery-eval
description: Evaluates a model's ability to retrieve relevant tables and text passages from a hybrid corpus to satisfy complex, multi-part analytical user requests (Data Product Requests). It probes multi-modal data integration and semantic clustering capabilities by requiring complete alignment between a request and its underlying data assets. Use when the user wants to benchmark on HybridQA, TAT-QA, ConvFinQA, or asks about evaluating this task. Reports Full Recall@100.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.21737
  bibtex_key: zhang2025dataproductbenchmark
  confidence: high
---

# data-product-discovery-eval

> From Factoid Questions to Data Product Requests: Benchmarking Data Product Discovery over Tables and Text — Zhang et al. (2025) (arXiv:2510.21737, 2025)

## What this evaluates

Evaluates a model's ability to retrieve relevant tables and text passages from a hybrid corpus to satisfy complex, multi-part analytical user requests (Data Product Requests). It probes multi-modal data integration and semantic clustering capabilities by requiring complete alignment between a request and its underlying data assets.

## Datasets

- **HybridQA** — total ?; splits: train (4889), dev (2008), test (1986)
- **TAT-QA** — total ?; splits: train (1157), dev (211), test (245)
- **ConvFinQA** — total ?; splits: train (2498), dev (426), test (875)

## Metrics

- `Recall@20` — range: [0, 1]
  - Fraction of ground-truth tables or passages retrieved within the top 20 results. Considers partial matches for individual tables or passages.
- `Recall@100` — range: [0, 1]
  - Fraction of ground-truth tables or passages retrieved within the top 100 results. Considers partial matches for individual tables or passages.
- `Full Recall@100` **(primary)** — range: [0, 1]
  - Binary metric indicating whether all ground-truth tables and passages for a data product are retrieved within the top 100 results. Considers only complete matches.

## Input / output format

**Input**: A natural-language Data Product Request (DPR) describing a complex analytical need requiring multiple tables and passages.

**Output**: A ranked list of retrieved tables and text passages from the corpus.

## Scoring recipe

```python
def compute_recall(retrieved_top_k, ground_truth, k=100):
    retrieved_set = set(retrieved_top_k[:k])
    gt_set = set(ground_truth)
    if not gt_set: return 0.0
    return len(retrieved_set & gt_set) / len(gt_set)

def compute_full_recall(retrieved_top_100, ground_truth):
    return 1.0 if set(ground_truth).issubset(set(retrieved_top_100)) else 0.0
```

## Common pitfalls

- Partial matches are acceptable for table/text Recall@K, but Full Recall@100 requires retrieving every single ground-truth table and passage for a data product.
- The benchmark relies on LLM-generated DPRs and LLM-as-a-judge validation, so evaluation results may reflect generation/judge biases rather than pure retrieval capability.
- Ground-truth alignments are pruned if judges flag tables as irrelevant, which can shrink the effective test set size per instance.

## Evidence (verbatim from paper)

> Table and text retreival alone consider partial matches, while under data product, Full Recall $@$ 100 considers only complete matches.

## Citation

```bibtex
@misc{zhang2025dataproductbenchmark,
  title={From Factoid Questions to Data Product Requests: Benchmarking Data Product Discovery over Tables and Text},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.21737}
}
```

- arXiv: 2510.21737