Research

Evaluates an agent's ability to perform logical deduction and multi-hop reasoning over long, noise-rich unstructured text without pre-defined schemas or tables. It probes the model's capacity to actively synthesize scattered evidence and filter out irrelevant distractors to arrive at a correct decision. Use when the user wants to benchmark on TACT, or asks about evaluating this task. Reports Exact Match (EM).

Evaluates the perceptual naturalness and quality of text-to-speech synthesis. It probes the model's ability to generate high-fidelity audio waveforms that are indistinguishable from human speech. Use when the user wants to benchmark on Internal US English Test Set, Custom 100-Sentence Test Set, News Headlines Test Set, or asks about evaluating this task. Reports MOS.

Evaluates code generation models on competition-level algorithmic programming problems. It probes fine-grained capabilities across different programming skills and difficulty levels by measuring whether generated Python programs correctly solve given problems under strict constraints. Use when the user wants to benchmark on TACO, or asks about evaluating this task. Reports pass@k.

This protocol re-evaluates tabular benchmarks to measure how validation strategy (holdout vs. 5-fold cross-validation) and hyperparameter optimization budgets affect model selection and reported performance. It probes the robustness of empirical conclusions in tabular machine learning when standard holdout validation is replaced with cross-validation ensembles. Use when the user wants to benchmark on TabZilla-hard, Grinsztajn et al. (2022) benchmark, or asks about evaluating this task. Report...

Evaluates computational extrapolation and algorithmic generalization in tabular learning models by testing their ability to predict target values outside the training distribution. It probes whether models learn statistical interpolation versus deterministic computation on program-verified synthetic math problems. Use when the user wants to benchmark on TabularMath, or asks about evaluating this task. Reports rounded consistency.

Evaluates the cross-dataset transferability of pretrained tabular generative models by measuring how well synthesized data preserves column distributions and pairwise correlations compared to ground truth tables. Use when the user wants to benchmark on Kaggle, GitTables, or asks about evaluating this task. Reports overall average.

Evaluates zero-shot and few-shot transfer learning capabilities of a language model on diverse tabular prediction tasks. It probes the model's ability to generalize across unseen datasets without fine-tuning, leveraging serialized row data and column headers to predict categorical or regression targets. Use when the user wants to benchmark on UniPredict Benchmark, Grinsztajn Benchmark, AutoML Multimodal Benchmark (AMLB), OpenML CC-18 Benchmark, OpenML CTR-23 Benchmark, or asks about evaluatin...

Evaluates the calibration and reliability of confidence scores produced by LLMs when answering questions over tabular data. It probes how well predicted confidence aligns with actual accuracy across different elicitation methods and dataset complexities. Use when the user wants to benchmark on WikiTableQuestions, TableBench, or asks about evaluating this task. Reports smooth ECE.

Evaluates large language models on predictive tabular tasks, including classification, regression, and missing value imputation. It probes the model's ability to reason over structured data, handle mixed numerical and textual features, and perform few-shot or long-context learning on tables. Use when the user wants to benchmark on Kaggle (Classification & Regression), Tabular Benchmark (Grinsztajn et al., 2022), or asks about evaluating this task. Reports ROC-AUC.

This evaluation probes a model's ability to perform few-shot in-context learning and standard classification on high-dimensional, heterogeneous tabular data. It specifically measures how well biaxial attention and meta-learning improve performance across medical, financial, and energy domains, and how robust the model is to varying support set sizes and selection strategies. Use when the user wants to benchmark on TALENT, OpenML-CC18, or asks about evaluating this task. Reports accuracy (ACC).

Evaluates feature selection methods by measuring downstream neural network performance on tabular datasets containing controlled extraneous features. It probes whether selected features improve or maintain predictive accuracy for classification and reduce error for regression tasks. Use when the user wants to benchmark on ALOI (AL), California Housing (CA), Covertype (CO), Eye Movements (EY), Gesture (GE), Helena (HE), Higgs 98k (HI), House 16K (HO), Jannis (JA), Otto Group Product Classifica...

Evaluates the effectiveness of deep image embedding clustering methods compared to traditional clustering algorithms on heterogeneous tabular datasets. It probes whether architectures designed for spatial image data can effectively learn representations for low-dimensional, non-spatial tabular data. Use when the user wants to benchmark on malware, mice, vehicle, olive, dermatology, breast cancer, Ecoli, or asks about evaluating this task. Reports clustering accuracy.

This evaluation probes the robustness and relative performance of tabular machine learning models when subjected to expert-level, dataset-specific preprocessing pipelines rather than standardized baselines. It specifically measures how feature engineering, hyperparameter optimization, and test-time adaptation shift model rankings and close performance gaps across real-world competition datasets. Use when the user wants to benchmark on Kaggle competition datasets (MBGM, BPCCM, HQC, SCTP, PSSDP...

Evaluates automated tabular data cleaning pipelines by measuring downstream classification accuracy and calibration when processed by a Tabular Foundation Model (TabPFN v2). It probes whether cleaning strategies can effectively align dirty data distributions with the model's learned prior to improve predictive performance. Use when the user wants to benchmark on OpenML CC18 Benchmark Suite (D1–D10), or asks about evaluating this task. Reports accuracy.

Evaluates the predictive performance and stability of 32 deep learning and tree-based tabular models across a large collection of diverse tabular datasets. It probes how well different architectures handle classification and regression tasks, and how dataset characteristics influence method rankings. Use when the user wants to benchmark on LAMDA-TALENT Benchmark, or asks about evaluating this task. Reports average_rank.

This protocol evaluates the faithfulness of feature attributions for LLM-based tabular classifiers. It measures how well an attribution method ranks features by sequentially masking them in importance order and tracking the resulting drop in the model's predicted class probability. Use when the user wants to benchmark on Adult Income, Heart Disease, or asks about evaluating this task. Reports faithfulness.

Evaluates deep learning models on table structure recognition (TSR) and table content recognition (TCR) by predicting LaTeX token sequences from tabular images. It probes the model's ability to accurately reconstruct table layouts and textual content under varying aspect ratios and sequence lengths. Use when the user wants to benchmark on TabLeX, or asks about evaluating this task. Reports EMA.

Probes multimodal large language models' ability to perform spatially grounded reasoning over complex hierarchical tables. It specifically evaluates performance degradation across three cognitive levels (Perception, Reasoning, Analysis) and measures how explicit spatial anchoring mitigates perceptual overload and spatial attention failure. Use when the user wants to benchmark on TableVision, or asks about evaluating this task. Reports exact-match Accuracy (%).

Evaluates table structure recognition (TSR) and cell detection capabilities by comparing two tokenization schemes (OTSL vs. HTML) on transformer-based image-to-sequence models. It measures how well the model predicts table layouts and cell bounding boxes across diverse document types. Use when the user wants to benchmark on PubTabNet, FinTabNet, PubTables-1M, or asks about evaluating this task. Reports Tree Edit Distance score (TEDs).

Evaluates graph-based machine learning models for sequence labeling (BIESO) and table row detection on handwritten historical register books. The protocol tests the models' ability to segment table rows and label cell boundaries using pre-extracted textline and column features rather than raw images. Use when the user wants to benchmark on Dataset1, Dataset2, or asks about evaluating this task. Reports F1 score.

Evaluates the ability of retrieval-augmented large language models to perform in-context learning on tabular data for classification and regression tasks. It probes how well non-parametric retrieval of support instances scales with dataset size and compares against numeric-based and classic tabular baselines. Use when the user wants to benchmark on Held-out Tabular Benchmark, or asks about evaluating this task. Reports AUROC, NMAE.

Evaluates the predictive performance of tabular machine learning models across 51 real-world datasets under standardized, reproducible protocols. It probes how hyperparameter tuning, nested cross-validation, and post-hoc ensembling affect peak performance and efficiency trade-offs. Use when the user wants to benchmark on TabArena, or asks about evaluating this task. Reports predictive performance.

Evaluates NLP models' ability to identify and mask personally identifiable information (direct and quasi-identifiers) in legal texts while preserving non-sensitive content. It probes both privacy protection (full coverage of masking spans) and information utility (minimizing unnecessary masking of non-identifying entities). Use when the user wants to benchmark on TAB corpus, or asks about evaluating this task. Reports ER_{di}.

Evaluates a unified text-to-text transformer's ability to generalize across diverse NLP tasks including language understanding, summarization, question answering, and machine translation. The protocol tests the efficacy of pre-training objectives, data scaling, and consistent text-to-text fine-tuning pipelines. Use when the user wants to benchmark on GLUE, SuperGLUE, CNN/Daily Mail, SQuAD, WMT, or asks about evaluating this task. Reports GLUE average score.