Research

Evaluates fine-grained temporal understanding on dense dynamic videos, probing camera motion, scene transitions, action sequences, and multi-subject interactions. It measures how well models capture dynamic visual elements and handle varying video complexities. Use when the user wants to benchmark on TUNA, or asks about evaluating this task. Reports F1 score, Accuracy.

This evaluation protocol probes the language understanding, reasoning, and instruction-following capabilities of Portuguese LLMs across diverse domains including academic exams, natural language inference, physical commonsense, and code generation. It is specifically designed to provide reliable training signals during pretraining and assess post-training alignment. Use when the user wants to benchmark on ARC Challenge, Calame, Global PIQA, HellaSwag, LAMBADA, ENEM, BLUEX, OAB, Belebele, MMLU...

Evaluates the ability of LLMs to improve reasoning performance at test time through self-reflection and targeted variant question synthesis, without external supervision. It probes how well a model can adapt its policy to difficult mathematical and general reasoning problems by diagnosing its own failures and generating corrective training signals. Use when the user wants to benchmark on AMC23, MATH-500, Minerva, OlympiadBench, AIME 2024, AIME 2025, GPQA-Diamond, MMLU-Pro, or asks about evalu...

Evaluates a text-to-speech model's ability to generate audio that matches natural language descriptions of speaker attributes (gender, accent, pitch, speaking rate, recording quality) and overall audio fidelity. It measures both objective acoustic metrics and subjective human ratings of relevance and naturalness. Use when the user wants to benchmark on MLS, LibriTTS-R, or asks about evaluating this task. Reports MOS.

Evaluates the impact of probabilistic versus deterministic duration modeling on the naturalness and intelligibility of non-autoregressive text-to-speech systems. It specifically probes how well stochastic duration predictors handle prosodic variability and disfluencies in spontaneous speech compared to read-aloud speech. Use when the user wants to benchmark on LJ, RS, TSGD2, AptS, or asks about evaluating this task. Reports CMOS.

This protocol evaluates a 3D convolutional auto-encoder for removing reverberation artifacts (clutter) from transthoracic echocardiographic (TTE) sequences. It measures how well the network preserves cardiac structures while suppressing simulated artifacts, using synthetic data with known ground truth for training and validation, and normal in-vivo sequences for testing. Use when the user wants to benchmark on Synthetic TTE sequences, or asks about evaluating this task. Reports reconstruction...

Evaluates the ability of models to detect human body forgeries generated by diffusion models. It probes spatiotemporal motion inconsistencies and generalization across different generation configurations and unseen manipulation models. Use when the user wants to benchmark on TT-DF, or asks about evaluating this task. Reports AUC.

Evaluates lesion detection performance on synthetic and real breast mammography/tomosynthesis images. Probes the model's ability to localize lesions across varying breast densities, lesion sizes, and lesion densities using a free-response receiver operating characteristic (FROC) framework. Use when the user wants to benchmark on T-SYNTH, EMBED, or asks about evaluating this task. Reports FROC (Sensitivity vs. Average False Positives per Image).

This benchmark evaluates an AI system's ability to perform fact verification using time-series evidence. It probes multi-timeframe temporal reasoning, cross-series numerical analysis, and the generation of factually consistent justifications aligned with human annotations. Use when the user wants to benchmark on TSVer, or asks about evaluating this task. Reports Accuracy.

Evaluates models on Time Series Extrinsic Regression (TSER), where the goal is to predict a single continuous scalar value from multivariate time series inputs of varying lengths and dimensions. It probes the model's ability to handle irregular time series, missing values, and diverse domain-specific patterns without imputation. Use when the user wants to benchmark on Monash TSER Archive, or asks about evaluating this task. Reports R2.

Evaluates a model's ability to perform sequential recommendation with a focus on capturing repeat-aware temporal patterns. It measures how well the model balances predicting new items versus recurring items based on user interaction history and time intervals. Use when the user wants to benchmark on RetailRocket, LastFM, Diginetica, or asks about evaluating this task. Reports HR@K.

Evaluates large language models on time series question answering across five tasks: forecasting, imputation, anomaly detection, classification, and open-ended reasoning. It probes the model's ability to integrate textual context with numerical time series data for both precise numerical prediction and natural language explanation. Use when the user wants to benchmark on TSQA, or asks about evaluating this task. Reports accuracy.

Evaluates the quality of discovered motif sets in time series by measuring alignment with ground truth segments. It accounts for variable-length patterns and time warping while penalizing both false discoveries and missed patterns. Use when the user wants to benchmark on TSMD benchmark datasets, or asks about evaluating this task. Reports F1-score.

Evaluates large language models' ability to understand and generate structured scene graphs from textual narratives. It probes spatial reasoning, action decomposition, and the capacity to map dynamic descriptions to discrete visual or structural elements. Use when the user wants to benchmark on TSG Bench, or asks about evaluating this task. Reports Exact Match (EM) / Accuracy, Precision, Recall, Macro F1.

Evaluates how time series foundation models scale in forecasting accuracy and uncertainty calibration as model size, compute, and training data size increase. It probes both in-distribution generalization and out-of-distribution transfer capabilities across multiple standard time series forecasting benchmarks. Use when the user wants to benchmark on Monash subset, LSF subset, or asks about evaluating this task. Reports NLL.

Evaluates object detection models on traffic surveillance footage under diverse weather conditions and varying degrees of vehicle occlusion. It probes robustness to environmental degradation, scale variation, and dense urban traffic scenarios. Use when the user wants to benchmark on TSBOW, or asks about evaluating this task. Reports mAP50.

Evaluates large language models' ability to perform time series analysis and reasoning across six tasks (anomaly detection, classification, characterization, comparison, data transformation, and temporal relationship) using three question formats (true-or-false, multiple-choice, and puzzling). Use when the user wants to benchmark on TSAQA, or asks about evaluating this task. Reports accuracy.

Evaluates the ability of large multimodal models to generate accurate, domain-agnostic natural language descriptions of time series trends. It probes cross-modal alignment between visual time series plots (or extracted features) and textual trend explanations. Use when the user wants to benchmark on TS-Insights, or asks about evaluating this task. Reports final_score.

Evaluates the forecasting accuracy and computational efficiency of deep learning models on multivariate time series data. It probes how architectural choices, preprocessing steps, and spatial-temporal processing configurations impact performance across varying forecasting horizons. Use when the user wants to benchmark on Weather, Solar-Energy, ECL, Traffic, or asks about evaluating this task. Reports MAE.

Evaluates the factual accuracy and truthfulness of large language models by measuring their ability to select correct answers over common misconceptions. It probes the model's capacity to resist generating plausible but false statements across diverse categories like health, law, and politics. The benchmark specifically tests whether models can identify and output factually correct responses when presented with multiple candidate answers. Use when the user wants to benchmark on TruthfulQA, or...

Evaluates an LLM's factual accuracy and hallucination mitigation across multiple-choice, short-form, and long-form generation tasks. It measures the trade-off between truthfulness and informativeness, and quantifies the exact number of supported versus unsupported facts in generated text. Use when the user wants to benchmark on TruthfulQA, BioGEN, or asks about evaluating this task. Reports Accuracy, True*Info, FActScore.

This benchmark evaluates a model's ability to perform true multimodal in-context learning by requiring it to solve tasks that depend on both visual and textual information from provided demonstrations. It probes whether models can correctly attend to and utilize visual context in few-shot examples rather than relying on superficial textual patterns or prior knowledge. Use when the user wants to benchmark on TrueMICL, or asks about evaluating this task. Reports accuracy.

Binary classification of truck driving risk on specific highway segments based on historical behavior, short-term trip dynamics, and real-time traffic conditions. It probes a model's ability to predict forward collision warning events using a small, highly imbalanced dataset of real-world trajectory data. Use when the user wants to benchmark on Truck Driving Risk Dataset, or asks about evaluating this task. Reports Accuracy.

This benchmark evaluates reading comprehension on complex, compositional trivia questions that require multi-sentence reasoning and handling high lexical variability. It tests a model's ability to locate and extract precise answers from large, noisy evidence documents across different domains. Use when the user wants to benchmark on TriviaQA, or asks about evaluating this task. Reports exact match (EM).