Research

Evaluates sequential recommendation models across accuracy, fairness, stability, and efficiency dimensions. It tests whether models can correctly rank items based on user interaction history and assesses their robustness, bias, and computational cost. Use when the user wants to benchmark on Yelp, ML-100K, Beauty, or asks about evaluating this task. Reports Recall@5.

Evaluates a model's ability to extract the smallest verbatim evidence block from a single tool observation given a focused query, while maximizing compression and correctly returning empty output for true negatives. Use when the user wants to benchmark on Squeez Benchmark, or asks about evaluating this task. Reports F1.

Evaluates a novel 'Squashing' activation function against standard alternatives (ReLU, Sigmoid, Tanh) on synthetic 2D classification tasks and the Fashion-MNIST image classification benchmark. It measures how well continuously differentiable logical approximations perform compared to conventional non-linearities in terms of convergence speed and final classification accuracy. Use when the user wants to benchmark on Fashion-MNIST, Synthetic 2D Classification, or asks about evaluating this task...

Evaluates long-document, question-focused summarization quality through structured human ratings and automatic metric correlation. It probes a model's ability to generate accurate, comprehensive, and high-quality summaries that align with human preferences rather than relying on surface-level n-gram overlap. Use when the user wants to benchmark on SQuALITY, or asks about evaluating this task. Reports Human Rating (1-100).

Probes a model's ability to perform extractive reading comprehension while correctly identifying when a question cannot be answered from the provided context. It forces models to distinguish between answerable and unanswerable questions, testing knowledge gap detection and resistance to semantically relevant distractors. Use when the user wants to benchmark on SQuAD 2.0, or asks about evaluating this task. Reports F1.

Extends extractive QA by allowing questions that have no answer in the passage, testing the model's ability to abstain or predict a null span. It evaluates robustness against unanswerable questions. Use when the user wants to benchmark on SQuAD v2.0, or asks about evaluating this task. Reports F1.

Measures extractive question answering capability by requiring the model to identify a text span in a passage that answers a given question. It tests precise token-level span prediction and contextual understanding. Use when the user wants to benchmark on SQuAD v1.1, or asks about evaluating this task. Reports exact-match (EM).

Evaluates a model's ability to extract and rank answer candidates from a given context for phrase-indexed question answering. It probes both the quality of candidate retrieval and the accuracy of final answer selection against gold spans. Use when the user wants to benchmark on SQuAD v1.1, FQuAD, or asks about evaluating this task. Reports exact-match.

Measures a model's ability to extract precise answer spans from a given context paragraph in response to a natural language question, testing reading comprehension and span prediction. Use when the user wants to benchmark on SQuAD 1.1/2.0, or asks about evaluating this task. Reports F1.

Evaluates an LLM's ability to generate correct SQL queries for transforming building energy data schemas. It measures how well different prompt strategies and iterative optimization handle complex schema mappings, pivoting, and aggregation in real-world smart building datasets. Use when the user wants to benchmark on Building Energy Data Transformation Benchmark, or asks about evaluating this task. Reports Execution Accuracy.

This benchmark evaluates a model's ability to translate natural language questions into correct SQL queries across diverse, real-world database schemas. It specifically probes the model's capacity to handle complex, multi-operation statements and cross-domain syntax structures that are often underrepresented in traditional benchmarks. Use when the user wants to benchmark on SQL-Synth, or asks about evaluating this task. Reports execution accuracy (EX).

This evaluation compares the interactive analytics performance of four SQL-on-Hadoop systems (Impala, Drill, Spark SQL, Phoenix) by measuring query response times and resource utilization. It characterizes how each system's optimizer and execution engine handle join orders, operator selection, and data scanning across different storage formats and scaling configurations. Use when the user wants to benchmark on Unspecified SQL workloads (text/parquet), or asks about evaluating this task. Repor...

Evaluates the ability of signal quality indices (SQIs) to predict downstream task performance on medical time series. It measures how well an SQI correlates with and separates high-quality from low-quality signal segments for specific tasks like R-peak detection and atrial fibrillation classification. Use when the user wants to benchmark on Glasgow University database (GUDb), MIT-BIH Atrial Fibrillation Database (MIT-BIH AF), Deepbeat test subset, or asks about evaluating this task. Reports o...

This benchmark evaluates the ability of computer vision models to estimate dense scene flow, optical flow, and stereo disparity at ultra-high resolutions with fine structural details. It specifically probes how well methods handle high-frequency textures, non-rigid motion, unmatched regions, and sky areas where traditional benchmarks often lack detail. Use when the user wants to benchmark on Spring, or asks about evaluating this task. Reports 1px outlier rate.

Evaluates the stability of safety alignment in text-to-image diffusion models after benign fine-tuning. It probes whether models suffer silent safety failures where utility remains high but safety degrades under distribution shifts like multilingual or domain-specific adaptation. Use when the user wants to benchmark on ViSU, I2P, RAB, or asks about evaluating this task. Reports R.

Probes video question answering capabilities, specifically focusing on temporal reasoning, action causality, counterfactual inference, and fine-grained motion understanding within professional sports contexts. Use when the user wants to benchmark on Sports-QA, or asks about evaluating this task. Reports accuracy.

Evaluates multi-object tracking performance in sports scenes, specifically probing a model's ability to maintain track identities under fast, variable-speed motion and highly similar player appearances. Use when the user wants to benchmark on SportsMOT, or asks about evaluating this task. Reports HOTA.

Evaluates video-language models' ability to recognize and report content encoded purely in temporal sequences of noise-like frames, probing their temporal pattern recognition and susceptibility to 'time-blindness' despite strong spatial reasoning. Use when the user wants to benchmark on SpookyBench, or asks about evaluating this task. Reports exact-match accuracy.

Evaluates dialog state tracking performance by measuring how accurately an agent predicts and maintains the current state of a multi-turn spoken conversation. Use when the user wants to benchmark on SpokenWOZ, or asks about evaluating this task. Reports Joint Goal Accuracy (JGA).

Evaluates the ability of speech-based models to perform mathematical reasoning directly from audio inputs. It probes three distinct capabilities: direct arithmetic computation, contextual word-problem solving across single- and multi-step scenarios, and application of domain-specific mathematical knowledge in spoken language. Use when the user wants to benchmark on Spoken-MQA, or asks about evaluating this task. Reports accuracy.

Evaluates conversational question answering models on both clean text and noisy ASR transcripts, measuring their ability to maintain performance under speech recognition errors and leverage data distillation techniques. Use when the user wants to benchmark on CoQA, Spoken-CoQA, or asks about evaluating this task. Reports F1.

Evaluates multimodal long-context reasoning and figure/table comprehension on scientific papers. Tests direct question answering with images, full paper context, and chain-of-thought retrieval capabilities. Use when the user wants to benchmark on SPIQA, or asks about evaluating this task. Reports L3Score.

This benchmark evaluates the robustness of text-to-SQL models when natural language questions contain real-world synonyms replacing schema-related terms. It probes whether models rely on rigid lexical matching or can generalize to paraphrased queries while preserving the underlying database schema and target SQL query. Use when the user wants to benchmark on Spider, Spider-Syn, or asks about evaluating this task. Reports exact matching accuracy.

Evaluates patch-level histopathology classification across four organ types (Skin, Colorectal, Thorax, Breast). It probes a model's ability to correctly identify tissue morphologies using both a central patch and its surrounding contextual patches. Use when the user wants to benchmark on SPIDER, or asks about evaluating this task. Reports accuracy.