Research

Evaluates the linguistic robustness of LLMs by measuring performance degradation when standard English prompts are transformed into 38 regional dialects and ESL varieties. It probes whether models maintain accuracy and instruction-following capabilities across non-standard linguistic variations. Use when the user wants to benchmark on MMLU, ARC, TruthfulQA, GSM8K, HellaSwag, WinoGrande, IFEval, AlpacaFarm, MT-Bench, or asks about evaluating this task. Reports accuracy.

Evaluates the statistical fidelity and practical utility of synthetic human trajectory generation models by measuring how well generated trajectories perform on downstream mobility tasks compared to real trajectories. It probes whether synthetic data can replace real data without performance degradation across recommendation, prediction, labeling, and simulation tasks. Use when the user wants to benchmark on Foursquare Tokyo (TKY), Foursquare Istanbul (IST), Foursquare New York City (NYC), or...

Evaluates the training efficiency and scalability of AlphaFold-like models on GPU clusters. It measures per-step execution time, overall wall-clock training duration, and convergence speed across different hardware configurations and optimization techniques. Use when the user wants to benchmark on OpenFold dataset, or asks about evaluating this task. Reports step time.

Evaluates a training-free iterative inference method for audio source separation. It probes the model's ability to progressively refine noisy audio mixtures (speech or music) by optimizing blending ratios across multiple inference steps without retraining or architectural changes. Use when the user wants to benchmark on VCTK-DEMAND, DNS Challenge v3, MUSDB18-HQ, or asks about evaluating this task. Reports PESQ, UTMOS, uSDR.

Evaluates the quality of automatically generated multilingual word sense disambiguation (WSD) training corpora by training a supervised WSD system (IMS) on them and measuring performance on standard WSD benchmark datasets. It probes whether synthetic sense-annotated data can match or exceed manually annotated corpora, particularly for low-resource languages. Use when the user wants to benchmark on Senseval-2, Senseval-3, SemEval-2007, SemEval-2013, SemEval-2015, or asks about evaluating this ...

This benchmark evaluates a model's ability to classify the intention of a natural language question relative to a database schema. It probes whether the model can distinguish between answerable queries, questions requiring external knowledge, ambiguous queries, grammatically invalid non-SQL questions, and questions unrelated to the schema. Use when the user wants to benchmark on TRIAGESQL, or asks about evaluating this task. Reports Macro F1.

Evaluates the ability of spatio-temporal graph neural networks to forecast traffic speed under normal and construction work zone disruption conditions. It probes how well models integrate heterogeneous work zone data to capture nonlinear spatio-temporal dependencies and maintain accuracy during significant traffic flow deviations. Use when the user wants to benchmark on Richmond, Tyson’s, or asks about evaluating this task. Reports MAE.

This evaluation protocol assesses a model's ability to forecast future traffic speeds on road networks under varying conditions, including the impact of construction workzones. It probes spatio-temporal dependency modeling by measuring prediction accuracy across multiple forecast horizons (15, 30, and 60 minutes) on real-world highway sensor data. Use when the user wants to benchmark on Tyson's Corner, Los-loop, PEMS-BAY, or asks about evaluating this task. Reports RMSE.

Evaluates spatiotemporal models' ability to localize traffic collision events in time and space, and to forecast network-level congestion and emissions. It probes multi-horizon forecasting accuracy and spatial-temporal coherence under simulated disruption scenarios. Use when the user wants to benchmark on NYC Broadway corridor, or asks about evaluating this task. Reports containment_performance.

Evaluates the ability of spatiotemporal GNN models to forecast future traffic flow (speed or occupancy) based on historical sensor data and road network topology. Use when the user wants to benchmark on METR-LA, PEMS-BAY, PeMS04, or asks about evaluating this task. Reports MAE.

Evaluates the accuracy of multi-modal trajectory forecasting models in predicting the final destination of traffic agents (pedestrians and vehicles) over a future time horizon. Use when the user wants to benchmark on SDD, InD, Argoverse, or asks about evaluating this task. Reports Minimum final displacement error.

Evaluates the ability of deep learning models to classify mobile network traffic into specific application categories (e.g., video, music, games) and background services using extracted packet flow features. It specifically probes the effectiveness of semi-supervised VAE-CNN architectures and model pruning techniques for resource-constrained edge devices. Use when the user wants to benchmark on Private Campus Network Dataset, or asks about evaluating this task. Reports Accuracy.

This evaluation probes a model's ability to detect objective corporate events in financial news and translate those detections into actionable, timely trading signals. It measures how effectively the detected events predict short-term stock price movements and generate excess returns compared to a market benchmark. Use when the user wants to benchmark on EDT, or asks about evaluating this task. Reports Winning Rate.

Evaluates a model's ability to generate aspect-specific summaries from clinical abstracts and accurately cite the supporting source sentences. It probes factual recall, conciseness, and traceability in a medical domain setting. Use when the user wants to benchmark on TracSum, or asks about evaluating this task. Reports Claim Recall.

Probes the capability of real-time tumor and surrogate localization in MRI-guided radiotherapy using 2D sagittal cine MRI sequences. It evaluates how well algorithms can track anatomical motion across varying frame rates and multi-vendor MRI-linac hardware under clinically relevant conditions. Use when the user wants to benchmark on TrackRAD2025, or asks about evaluating this task. Reports tracking performance.

Evaluates the ability of deep learning models to detect and track high-speed, tiny objects (tennis and badminton balls) in broadcast sports videos. It probes robustness to motion blur, occlusion, and domain shifts by comparing single-frame vs. multi-frame tracking and transfer learning across different sports. Use when the user wants to benchmark on Tennis, Badminton, or asks about evaluating this task. Reports F1-measure.

Probes a GNN's capability to perform edge scoring on highly sparse, irregular scientific graphs by predicting the probability that a directional connection between two 3D space-point measurements originates from the same particle. Use when the user wants to benchmark on TrackML, or asks about evaluating this task. Reports edge probability.

Evaluates a model's ability to track objects through appearance-changing state transformations and explicitly model those transformations as a state graph. It probes spatiotemporal continuity, zero-shot object recovery, and semantic reasoning about object interactions. Use when the user wants to benchmark on VOST, VSCOS, M3-VOS, DAVIS 2017, VOST-TAS, or asks about evaluating this task. Reports Jaccard (J).

This benchmark evaluates an LLM's ability to detect and classify reward hacking behaviors in multi-turn code generation trajectories. It specifically probes contrastive anomaly detection capabilities by presenting clusters of mixed benign and malicious trajectories, testing whether models can disentangle subtle semantic and syntactic exploit patterns without prior taxonomy exposure. Use when the user wants to benchmark on TRACE, or asks about evaluating this task. Reports Detection Rate.

This benchmark evaluates a model's ability to predict post-click Gross Merchandise Volume (GMV) under delayed feedback conditions. It specifically probes how well models adapt to rapidly evolving label distributions through online streaming training and whether they can effectively handle the distinct statistical properties of single-purchase versus repurchase transactions. Use when the user wants to benchmark on TRACE, or asks about evaluating this task. Reports AUC.

This protocol evaluates training-free partial audio deepfake detection by analyzing the temporal continuity of frozen speech foundation model embeddings. It probes a model's ability to detect splice boundaries and synthetic insertions in speech without requiring labeled training data or architectural modifications. Use when the user wants to benchmark on PartialSpoof, HalfTruth Audio Deepfake (HAD), ADD 2023 Track 2, LlamaPartialSpoof, or asks about evaluating this task. Reports EER.

Evaluates the quality and efficiency of trace encoding methods for process mining event logs. It probes how well encodings preserve trace similarities (expressivity), their computational cost as data scales (scalability), and their suitability for downstream process mining tasks. Use when the user wants to benchmark on Process Mining Event Log Scenarios (1-5), or asks about evaluating this task. Reports T4.

Evaluates a model's ability to classify OpenTelemetry workflow traces as benign, suspicious, or malicious, and assesses its knowledge of cybersecurity frameworks via multiple-choice questions. Use when the user wants to benchmark on OpenTelemetry Workflow Traces, or asks about evaluating this task. Reports Overall Accuracy.

Evaluates large language models' ability to perform multi-table question answering across varying context lengths (8K–64K tokens) and complex reasoning tasks. It probes cross-table inference, symbolic reasoning, and handling of real-world relational data without Wikipedia bias. Use when the user wants to benchmark on TQA-Bench, or asks about evaluating this task. Reports accuracy.