Modeltables Eval

---
name: modeltables-eval
description: Evaluates the ability of retrieval systems to find relevant AI model tables from a heterogeneous corpus. It probes semantic understanding of structured data and cross-source table matching. Use when the user wants to benchmark on ModelTables, or asks about evaluating this task. Reports P@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.16106
  bibtex_key: [REDACTED]
  confidence: high
---

# modeltables-eval

> ModelTables: A Corpus of Tables about Models — Dong et al. (2025) (arXiv:2512.16106, 2025)

## What this evaluates

Evaluates the ability of retrieval systems to find relevant AI model tables from a heterogeneous corpus. It probes semantic understanding of structured data and cross-source table matching.

## Datasets

- **ModelTables** — total ?; splits: test (-1); repo https://github.com/RJMillerLab/ModelTables

## Metrics

- `P@1` **(primary)** — range: percent
  - Precision at 1: returns 1 if the top-ranked retrieved table is relevant to the query, and 0 otherwise. Averaged across all queries.

## Input / output format

**Input**: A query table (or model identifier) and a corpus of tables extracted from Hugging Face model cards, GitHub READMEs, arXiv papers, and Semantic Scholar.

**Output**: A ranked list of retrieved tables.

## Scoring recipe

```python
def compute_p_at_1(retrieved_tables, relevant_tables):
    top_1 = retrieved_tables[0]
    return 1.0 if top_1 in relevant_tables else 0.0

# Average over all queries
scores = [compute_p_at_1(q_retrieved, q_relevant) for q_retrieved, q_relevant in queries]
metric_value = sum(scores) / len(scores) * 100
```

## Common pitfalls

- Heterogeneous table formats across sources (e.g., transposed layouts, merged headers) can break naive retrieval without augmentation.
- S2ORC tables are raw unstructured text requiring LLM reconstruction, introducing potential parsing noise.
- Placeholder arXiv links in Hugging Face templates must be explicitly filtered to avoid irrelevant tables.

## Evidence (verbatim from paper)

> It establishes multi-source ground truth for table relatedness via model card links, paper citations, and shared datasets, enabling evaluation of table search methods. Empirical results show table-based dense retrieval (66.5% P@1) outperforms union-based semantic retrieval (54.8% P@1) and metadata-hybrid retrieval (54.1%), highlighting the need for better semantic understanding of structured model knowledge.

## Citation

```bibtex
@misc{dong2025modeltables,
  title={ModelTables: A Corpus of Tables about Models},
  author={Dong et al. (2025)},
  year={2025},
  note={arXiv:2512.16106}
}
```

- arXiv: 2512.16106