Research

This benchmark evaluates the cross-lingual transfer capability of neural news recommenders. It probes how well models trained monolingually on English news can generate accurate recommendations in 14 other languages under zero-shot and few-shot settings, with and without bilingual user consumption patterns. Use when the user wants to benchmark on xMIND, or asks about evaluating this task. Reports AUC.

Evaluates multimodal large language models on ultra-high-resolution remote sensing imagery using vision-language question answering. It probes both perception (e.g., object classification, counting, spatial relations) and reasoning capabilities across various sub-tasks. Use when the user wants to benchmark on XLRS-Bench, LRS-VQA, or asks about evaluating this task. Reports accuracy.

Evaluates the ability of text-to-SQL models to generate correct SQL or GQL queries for natural language questions across relational and graph databases. It measures execution accuracy by comparing the runtime results of generated queries against reference queries on specific database instances. Use when the user wants to benchmark on Spider, Bird, SQL-Eval, NL2GQL, or asks about evaluating this task. Reports Execution Accuracy (EX).

Evaluates a Text-to-SQL framework's ability to generate correct and efficient SQL queries from natural language questions across complex, cross-domain databases. It probes schema filtering, multi-generator candidate creation, and selection robustness. Use when the user wants to benchmark on BIRD, Spider, or asks about evaluating this task. Reports Execution Accuracy (EX).

Evaluates cross-lingual transfer capabilities of pre-trained language models across 11 diverse natural language understanding and generation tasks spanning over 100 languages. It measures how well models fine-tuned on English can generalize to zero-shot testing in other languages. Use when the user wants to benchmark on XGLUE, or asks about evaluating this task. Reports accuracy.

Evaluates the ability of graph neural networks to detect fraudulent transactions in large-scale, highly imbalanced e-commerce transaction graphs. It probes model performance under extreme class imbalance and measures the trade-off between detection accuracy, inference speed, and scalability across different graph sizes and distributed settings. Use when the user wants to benchmark on eBay-xlarge, eBay-large, eBay-small, or asks about evaluating this task. Reports AUC.

Evaluates a model's ability to detect violent events in untrimmed audio-visual videos under weak supervision. It probes the model's capacity to fuse complementary audio and visual cues to localize violence in class-imbalanced, long-range video sequences. Use when the user wants to benchmark on XD-Violence, or asks about evaluating this task. Reports AP.

Evaluates the ability of multi-agent LLM frameworks to automatically fix buggy Ruby code using test-driven feedback loops. It probes iterative code repair, self-reflection, and test generation capabilities under varying difficulty levels and error types. Use when the user wants to benchmark on xCodeEval (Ruby subset), or asks about evaluating this task. Reports pass@1.

Evaluates large language models on multilingual code understanding, generation, translation, and retrieval across 11 programming languages. It probes the model's ability to produce executable, correct code by validating outputs against unit tests rather than relying on lexical overlap. Use when the user wants to benchmark on xCodeEval, or asks about evaluating this task. Reports pass@5.

Evaluates a model's ability to perform cross-cultural machine translation, specifically probing its capacity to accurately transcreate culturally nuanced entity names across multiple language pairs rather than merely transliterating or omitting them. Use when the user wants to benchmark on XC-Translate, WMT (17-21), or asks about evaluating this task. Reports M-ETA.

Evaluates a machine learning model's ability to predict local chemical descriptors (oxidation state and coordination number) from experimental X-ray absorption spectra, specifically testing how well spectral domain mapping bridges the gap between simulated training data and real experimental measurements. Use when the user wants to benchmark on Combinatorial Zinc Titanate Thin Film XANES, or asks about evaluating this task. Reports OS/CN prediction accuracy.

Evaluates the reliability and operational suitability of white-box explainable AI methods (DeepLift, Integrated Gradients, LRP) when applied to deep neural network-based intrusion detection systems. It probes how well these methods preserve model accuracy, maintain consistency under repeated runs, resist adversarial noise, and compute efficiently across real-world network traffic datasets. Use when the user wants to benchmark on NSL-KDD, RoEduNet-SIMARGL2021, CICIDS-2017, or asks about evalua...

Evaluates the computational efficiency and explanation fidelity of an ROI-guided perturbation framework for medical image segmentation. It measures how effectively the method reduces computation while preserving segmentation accuracy and explanation quality compared to full occlusion baselines. Use when the user wants to benchmark on FLARE22, SAROS, CHAOS, or asks about evaluating this task. Reports Dice coefficient.

Evaluates LLM-based agents' ability to comprehend multilingual shopping instructions and successfully navigate interactive web environments across 14 languages. Use when the user wants to benchmark on X-WebAgentBench, or asks about evaluating this task. Reports Task Score.

Probes the spatial feature flow and patch-level influence in Vision Mamba models using classical control theory. It quantifies how input image patches drive hidden state dynamics across hierarchical layers, revealing domain-specific diagnostic feature extraction patterns. Use when the user wants to benchmark on CMMD, DermaMNIST, BloodMNIST, or asks about evaluating this task. Reports influence score.

This benchmark evaluates multilingual topic classification on social media tweets across four languages (English, Spanish, Japanese, Greek). It probes models' ability to generalize across languages and training regimes, including zero-shot, few-shot, monolingual, cross-lingual, and multilingual fine-tuning settings. Use when the user wants to benchmark on X-Topic, or asks about evaluating this task. Reports macro-F1.

Evaluates multi-modal large language models on progressive clinical reasoning in ophthalmic diagnosis. It tests the model's ability to perform a six-stage diagnostic chain (from image quality assessment to clinical decision-making) while integrating cross-modality imaging data and calibrating its uncertainty. Use when the user wants to benchmark on X-PCR, or asks about evaluating this task. Reports Stage-Wise Accuracy (SWA).

Evaluates the text rendering, text-to-image generation, and image understanding capabilities of a discrete autoregressive image generation model trained with reinforcement learning. It probes the model's ability to follow complex instructions, render long texts accurately, and generate high-fidelity images without relying on classifier-free guidance. Use when the user wants to benchmark on OneIG-Bench, LongText-Bench, DPG-Bench, GenEval, POPE, GQA, MMBench, SEEDBench-Img, DocVQA, OCRBench, or...

Evaluates an end-to-end navigation model's ability to predict robot dynamics and successfully navigate through structured and cluttered warehouse environments. It probes both open-loop trajectory and speed prediction accuracy, as well as closed-loop mission success, navigation efficiency, and motion smoothness in seen and out-of-distribution settings. Use when the user wants to benchmark on X-Mobility Warehouse Dataset, or asks about evaluating this task. Reports mission success rate (SR).

Evaluates large language models' ability to understand and classify disruptive weather impacts from historical and modern newspaper articles, and to answer related questions by ranking relevant information. It specifically probes models' capacity to handle climate-related polysemy, extract nuanced societal responses, and correctly identify passages without weather impacts. Use when the user wants to benchmark on WXImpactBench, or asks about evaluating this task. Reports F1-score.

This evaluation probes a model's ability to perform single-channel speech separation by learning discriminative time-frequency embeddings that group mixture components into distinct speaker clusters. It specifically tests generalization to unseen speakers and scaling to three-speaker mixtures without retraining. Use when the user wants to benchmark on WSJ0-based Speech Mixtures, or asks about evaluating this task. Reports SDR improvement (dB).

Evaluates speech recognition models on their ability to accurately transcribe spoken audio into text (WER) and characters (LER). It probes the effectiveness of unsupervised pre-training on raw audio for downstream acoustic modeling and decoding. Use when the user wants to benchmark on TIMIT, WSJ, or asks about evaluating this task. Reports WER.

Evaluates Word Sense Induction (WSI) by clustering contextualized word embeddings to predict sense assignments. It probes a model's ability to capture lexical polysemy and contextual meaning without supervised sense labels, using natural corpus distributions rather than artificially skewed benchmarks. Use when the user wants to benchmark on SemCor, or asks about evaluating this task. Reports F-B^3.

Evaluates whole-slide image (WSI) classification performance using self-supervised patch representations and feature-space data augmentation. It probes how well distribution-guided representation learning captures discriminative histopathological patterns for diagnostic subtyping. Use when the user wants to benchmark on USTC-EGFR, TCGA-EGFR, TCGA-LUNG-3K, or asks about evaluating this task. Reports micro-average area under the curve (AUC).