Research

Evaluates few-shot text classification on real-world tasks with class imbalance and long inputs. It probes a model's ability to leverage limited labeled examples, domain knowledge, and open-domain retrieval to classify text without a validation set. Use when the user wants to benchmark on RAFT, or asks about evaluating this task. Reports macro-F1.

Evaluates interactive 3D medical image segmentation by measuring how well models segment target structures under varying human-in-the-loop prompting strategies (points, boxes, scribbles) and iterative refinement protocols. It specifically probes the trade-off between interaction effort and segmentation accuracy across 2D and 3D architectures. Use when the user wants to benchmark on RadioActive, or asks about evaluating this task. Reports Dice.

Evaluates the transferability of self-supervised learning representations for classifying radio astronomy sources from interferometric cutout images. It benchmarks multiple SSL pretraining methods against ImageNet baselines using linear probing and full fine-tuning on downstream classification tasks. Use when the user wants to benchmark on MiraBest, RGZ, MSRS, VLASS, or asks about evaluating this task. Reports classification accuracy.

Probes the transfer learning capability of self-supervised vision models on radio astronomy morphology classification tasks across heterogeneous imaging pipelines, telescopes, and label granularities. Use when the user wants to benchmark on MiraBest, LoTSS DR2, Radio Galaxy Zoo DR1, or asks about evaluating this task. Reports accuracy.

Evaluates the accuracy of a numerical vector radiative transfer code by comparing its outputs for scattering phase functions, polarization, and albedo against established reference results from prior literature. Use when the user has predictions and gold and needs to compute disk-integrated total degree of polarization.

Evaluates the training efficiency and novel-view synthesis or reconstruction quality of neural radiance field methods by reducing the number of rays sampled during volume rendering. It measures how well adaptive ray allocation preserves rendering accuracy while accelerating convergence across diverse 3D scene benchmarks. Use when the user wants to benchmark on Realistic Synthetic 360°, Light Field (LF), LLFF, Tanks and Temples (T&T), Real-World 360°, DTU, or asks about evaluating this task. R...

Evaluates a vision-language model's ability to perform joint radiological diagnosis, abnormality detection, and multi-target segmentation on X-ray and CT images. It probes the model's capacity for open-ended visual question answering, precise pixel-level mask generation, and robustness to label-imbalanced medical data. Use when the user wants to benchmark on RadDiagSeg-D, VQA-RAD, SLAKE, or asks about evaluating this task. Reports F1, Dice.

Evaluates the robustness of image anomaly detection models against real-world imaging distortions, including free viewpoints, uneven illumination, and motion blur. It measures how well unsupervised and zero-shot methods localize and classify anomalies on industrial work platforms with foreign objects. Use when the user wants to benchmark on RAD, or asks about evaluating this task. Reports AUROC.

Probes multi-sport vision capabilities by evaluating unified ball tracking, racket pose estimation, and dynamic trajectory prediction across table tennis, tennis, and badminton. It tests both static perception and temporal modeling of human-object interactions. Use when the user wants to benchmark on RacketVision, or asks about evaluating this task. Reports ball tracking.

Evaluates high-speed autonomous driving capabilities including precise localization, long-range object detection and tracking, and robust mapping/SLAM under extreme dynamic conditions (up to 170 mph). The protocol benchmarks how well models maintain accuracy and latency when processing multi-modal sensor data at racing speeds where motion blur, sensor dropout, and rapid ego-motion are prevalent. Use when the user wants to benchmark on RACECAR, or asks about evaluating this task. Reports Avera...

Evaluates the ability to detect incorrect answers in chain-of-thought reasoning by analyzing inconsistencies across multiple reasoning paths. It measures how well intermediate steps can predict the correctness of a final answer without relying solely on the answer itself. Use when the user wants to benchmark on R2PE, or asks about evaluating this task. Reports Discernibility Score (DS).

Evaluates retrieval models on reasoning-driven medical tasks where document relevance is determined by alignment with inferred clinical diagnoses or multi-step reasoning paths rather than lexical or semantic overlap. Covers three task types—Q&A reference, clinical evidence, and clinical case retrieval—spanning eight medical sub-domains. Use when the user wants to benchmark on R2MED, or asks about evaluating this task. Reports nDCG@10.

Probes LLMs' ability to autonomously generate and execute code for complex reasoning and planning tasks across logic, spatial, order, optimization, search, and math domains. It evaluates how well models can iteratively explore, optimize, and self-check solutions in multi-turn code execution environments. Use when the user wants to benchmark on SymBench, Big-Bench-Hard, Reasoning-Gym, or asks about evaluating this task. Reports exact match or constraint check.

Evaluates a model's ability to perform unsupervised anomaly detection on multi-agent traffic trajectories in urban environments. It probes frame-wise recognition of rare and abnormal driving behaviors, including both individual maneuvers and context-dependent interactions between agents and static map features. Use when the user wants to benchmark on R-U-MAAD, or asks about evaluating this task. Reports frame-wise anomaly detection accuracy.

Evaluates the semantic alignment and retrieval capability of scene graph generation models and image-scene graph similarity frameworks. It measures how well generated or ground-truth scene graphs can retrieve corresponding images compared to other images in a dataset. Use when the user wants to benchmark on Visual Genome, Open Images V6, or asks about evaluating this task. Reports R-Precision@K.

Evaluates the accuracy of a regression framework (R-ICE) in estimating prompt-level inference carbon and energy emissions for LLMs using only token counts and publicly available performance data. It probes whether runtime can be reliably modeled as a piecewise linear function of input and output tokens without intrusive monitoring or architecture details. Use when the user wants to benchmark on HELM, or asks about evaluating this task. Reports average prediction error.

Evaluates complex reasoning capabilities of LLMs and MLLMs on graduate-level, multi-disciplinary academic questions in both English and Chinese. It probes the models' ability to handle rigorous, curriculum-based problems requiring extended chain-of-thought reasoning. Use when the user wants to benchmark on R-Bench-T, R-Bench-M, or asks about evaluating this task. Reports Top-1 accuracy.

This evaluation protocol assesses the speech synthesis and tokenization capabilities of the Qwen3-TTS model. It probes zero-shot voice cloning, multilingual and cross-lingual generation, controllable voice design, and long-form audio stability across multiple languages. Use when the user wants to benchmark on CommonVoice & Fleurs, LibriSpeech test-clean, Seed-TTS test set, TTS multilingual test set, CV3-Eval, InstructTTSEval, or asks about evaluating this task. Reports WER.

This protocol evaluates large language models across core competencies including general knowledge, reasoning, coding, and mathematics. It also assesses multilingual understanding, instruction following, and alignment with human preferences. Use when the user wants to benchmark on MMLU, MMLU-Pro, GPQA, HumanEval, GSM8K, MT-Bench, IFEval, or asks about evaluating this task. Reports accuracy.

Evaluates the reliability and accuracy of crowdsourced annotations for Quranic recitation audio. It measures annotator performance against expert labels, assesses inter-rater consistency, and validates an automated label-selection algorithm. Use when the user wants to benchmark on Quranic Audio Dataset, or asks about evaluating this task. Reports Matthews Correlation Coefficient (MCC).

Evaluates a model's ability to recommend relevant quotes for a given writing context. It probes cross-lingual recommendation capabilities across English, Standard Chinese, and Classical Chinese by measuring how accurately and highly a model ranks the correct quote among a pool of candidates. Use when the user wants to benchmark on QuoteR, or asks about evaluating this task. Reports MRR.

Evaluates time series forecasting models on a regime-balanced benchmark stratified by trend, seasonality, and forecastability. It probes a model's ability to handle varying context lengths, forecast horizons, and multivariate dependencies while mitigating domain bias and information leakage. Use when the user wants to benchmark on QuitoBench, or asks about evaluating this task. Reports MAE.

This benchmark evaluates a model's ability to classify high-energy physics particle jets as originating from quarks or gluons using pixelized detector data. It probes feature extraction and binary classification performance across different input channel configurations and jet transverse momentum ranges. Use when the user wants to benchmark on Simulated CMS LHC Jet Data (DELPHES), or asks about evaluating this task. Reports AUC.

Evaluates a model's ability to fact-check real-world numerical claims containing statistical and temporal expressions. It probes evidence retrieval, claim decomposition, and natural language inference to predict veracity (True, False, or Conflicting). Use when the user wants to benchmark on NumTemp, or asks about evaluating this task. Reports Macro-F1.