Data & Analytics

Evaluates the capability of physics-informed autoregressive models to accurately forecast time-dependent partial differential equations and global atmospheric variables over multi-step horizons. Use when the user wants to benchmark on PDE Benchmarks (Wave, Reaction, Convection, Heat), ERA5, or asks about evaluating this task. Reports RMSE.

Evaluates the ability of machine learning and statistical models to forecast daily patient flows at urgent care clinics, specifically testing robustness to concept drift induced by pandemic disruptions using quasi-real-time proxy variables. Use when the user wants to benchmark on Clinic 1 & 2 patient flow data, or asks about evaluating this task. Reports MAPE.

Evaluates machine learning models' ability to detect malware and anomalies in data center compute nodes using high-resolution power consumption data and hardware performance counters. It probes real-time anomaly detection capabilities under severe class imbalance and varying computational workloads. Use when the user wants to benchmark on pAElla Malware & Benchmark Dataset, or asks about evaluating this task. Reports weighted F1-score.

Evaluates the end-to-end latency and energy efficiency of CPU/GPU scheduling strategies for agentic AI workloads under batched request arrivals. Use when the user has predictions and gold and needs to compute P50 latency.

Evaluates the ability of statistical and machine learning models to detect anomalies in battery discharge capacity profiles across different chemistries. It probes cross-chemistry generalization and model robustness on imbalanced, rare-anomaly datasets typical of electrochemical systems. Use when the user wants to benchmark on MIT/Stanford (Severson), Tohoku, or asks about evaluating this task. Reports AUROC.

Evaluates the ability of an encoder-decoder LSTM model combined with statistical hypothesis testing to detect unexpected anomalies in optical network quality-of-transmission metrics. It probes whether predicted soft-failure trajectories can distinguish predictable degradation from sudden, anomalous BER deviations in real-time. Use when the user wants to benchmark on Synthetic Optical Network PLM Dataset, or asks about evaluating this task. Reports Accuracy.

This evaluation probes a model's ability to correctly classify instances into known classes while accurately detecting and rejecting instances from unknown, unseen classes. It measures both outlier detection capability via ROC analysis and multi-class recognition performance including an explicit unknown category. Use when the user wants to benchmark on MNIST, MS Challenge, Android Genome, or asks about evaluating this task. Reports AUC.

Evaluates the capability of financial LLMs and agents across seven core financial task categories, including information extraction, sentiment analysis, question answering, text generation, risk management, forecasting, and decision-making. The benchmark aggregates 42 existing financial datasets to provide a standardized comparison of model performance and compliance readiness. Use when the user wants to benchmark on Open FinLLM Leaderboard (42 financial datasets), or asks about evaluating th...

This evaluation probes the ability of machine learning and deep learning models to classify malware into specific families based on their assembly-level instruction sequences (opcodes). It compares traditional feature-engineering approaches using 1-gram and 2-gram n-grams against an end-to-end 1D CNN that processes raw opcode sequences. Use when the user wants to benchmark on OpCode Malware Dataset, or asks about evaluating this task. Reports Accuracy.

Evaluates geospatial anomaly detection models on synthetic human mobility data. It probes the ability of algorithms to identify injected anomalous movement patterns across different granularities (staypoint, trip, agent) while controlling for demographic, temporal, and spatial factors. Use when the user wants to benchmark on NUMOSIM, or asks about evaluating this task. Reports Average Precision (AP).

Evaluates machine learning models for predicting Critical Heat Flux (CHF) in nuclear thermal-hydraulics, probing their ability to capture complex, multi-regime physical behaviors and produce well-calibrated, informative uncertainty estimates across different flow regimes. Use when the user wants to benchmark on NRC dataset, or asks about evaluating this task. Reports RMSPE.

Evaluates recommendation algorithms for predicting drug-target and drug-disease interactions. It specifically probes the model's ability to handle bidirectional drug effects (therapeutic vs. adverse) and rank candidate pairs accurately under cold-start cross-validation scenarios. Use when the user wants to benchmark on Drug-Protein benchmark dataset, Drug-Disease benchmark dataset, or asks about evaluating this task. Reports AUPR.

Evaluates machine learning models' ability to predict ground-level NO2 concentrations in urban areas using multi-source environmental and demographic data. Probes spatial-temporal regression capabilities and model generalization across different cities and time periods. Use when the user wants to benchmark on CityAQVis Urban NO2 Dataset, or asks about evaluating this task. Reports R2 Score.

Evaluates transformer models' ability to classify news articles as biased or unbiased, comparing standard fine-tuning against domain-adapted training. It further probes model decision-making by analyzing word-level SHAP attribution magnitudes and lexical feature importance across true and false predictions. Use when the user wants to benchmark on BABE, or asks about evaluating this task. Reports Binary F1.

Evaluates automated machine learning strategies for supervised learning on multimodal tabular datasets containing text, numeric, and categorical features. It probes how well different featurization methods, neural backbones, and ensemble aggregation techniques handle mixed data types and extract predictive signal from text fields. Use when the user wants to benchmark on Multimodal Tabular Benchmark (18 datasets), or asks about evaluating this task. Reports accuracy.

Evaluates machine learning models on hierarchical network traffic classification tasks, including top-level protocol identification and malware detection, as well as mid-level application and malware type classification. Use when the user wants to benchmark on VPN-nonVPN + $(\mathsf{Net})^2$ + CICIDS2017, or asks about evaluating this task. Reports Macro-average F1 score.

Evaluates zero-shot and fine-tuned time series foundation models against traditional statistical and machine learning baselines for predicting age- and country-specific mortality rates over 5, 10, and 20-year horizons. Use when the user wants to benchmark on Global Mortality Rates (50 countries, 111 age groups), or asks about evaluating this task. Reports SMAPE.

Evaluates the precision and accuracy of stellar flux recovery and positional measurements in simulated Roman Space Telescope images. It probes how well effective PSF models can recover input fluxes and coordinates across different spatial grids, filters, and magnitudes. Use when the user has predictions and gold and needs to compute modified median absolute deviation ($\hat{\sigma}$).

Evaluates the inference latency and end-to-end turn-around time of five machine learning model-serving frameworks across four distinct real-world inference scenarios. It probes how framework specialization (DL-specific vs. general-purpose) and input payload size affect serving performance and stability. Use when the user wants to benchmark on Malware detection, Cryptocoin price forecasting, Image classification, Sentiment analysis, or asks about evaluating this task. Reports average_latency.

This evaluation probes the hardware efficiency and resource constraints of training deep learning models on mobile SoCs. It measures how different model architectures and batch sizes impact GPU/CPU utilization, power/energy draw, and memory footprint during training. Use when the user has predictions and gold and needs to compute GPU utilization.

Evaluates the geometric alignment between visual and textual attention key vectors in multimodal LLMs. It probes whether visual inputs occupy an out-of-distribution subspace relative to the text-centric key space learned during pretraining. Use when the user has predictions and gold and needs to compute MMD.

Evaluates the training performance and scalability of machine learning implementations across diverse hardware and software stacks by measuring time to solution under standardized model and hyperparameter configurations. Use when the user wants to benchmark on MLPerf Training Suite, or asks about evaluating this task. Reports time to solution.

Evaluates the energy efficiency of machine learning systems across diverse hardware scales (data center, edge, tiny) and workloads (inference and training). It measures how effectively systems convert electrical energy into computational progress, tracking improvements in samples processed per joule over time and across system configurations. Use when the user wants to benchmark on MLPerf, or asks about evaluating this task. Reports samples per joule.

This evaluation protocol measures the throughput and latency of machine learning input data pipelines across standard computer vision and NLP benchmarks. It probes how efficiently a data processing framework can ingest, transform, and feed batches to a training loop compared to sequential baselines and competing systems. Use when the user wants to benchmark on ImageNet, COCO, WMT16, WMT17, or asks about evaluating this task. Reports epoch duration.