Research

Evaluates zero-shot cross-lingual transfer of multilingual language models. Models are trained exclusively on English-labeled data and then tested on 40 typologically diverse languages across nine tasks spanning sentence classification, structured prediction, question answering, and sentence retrieval. Use when the user wants to benchmark on XTREME, or asks about evaluating this task. Reports accuracy.

Evaluates an LLM's ability to perform context-aware, instruction-guided revisions of academic paper sections. It probes controllable editing capabilities, specifically measuring adherence to revision instructions, clarity, conciseness, and alignment with scientific writing standards through automated pairwise comparisons and human scoring. Use when the user wants to benchmark on XtraQA, or asks about evaluating this task. Reports Length-controlled (LC) win rate.

Evaluates cross-task semantic consistency in unified multimodal models by measuring how well generation and understanding tasks align on shared scene-graph facts. It probes whether architectural unification leads to representation-level coherence or merely independent task accuracy, specifically highlighting failures like consistent hallucination. Use when the user wants to benchmark on XTC-Bench, or asks about evaluating this task. Reports CCTA, AW-CCTA.

Evaluates a model's ability to perform extreme abstractive summarization by generating a single-sentence summary from a full news article, requiring synthesis, paraphrasing, and inference across document sections. Use when the user wants to benchmark on XSum, or asks about evaluating this task. Reports automatic metrics.

Probes whether large language models exhibit exaggerated safety behaviors by refusing safe prompts due to lexical overfitting or system prompt effects. It measures the model's ability to distinguish between genuinely unsafe requests and safe prompts that merely resemble unsafe content. Use when the user wants to benchmark on XSTest, or asks about evaluating this task. Reports response_classification.

Evaluates the fidelity and stability of state-explaining methods in Reinforcement Learning across tabular and image-based environments. It measures how accurately explanations identify critical states and how consistently they perform under perturbations. Use when the user wants to benchmark on XRL-Bench Environments (DunkCityDynasty-v1, LunarLander-v2, CartPole-v0, FlappyBird-v0, Breakout-v0, Pong-v0), or asks about evaluating this task. Reports AIM.

Evaluates the capability of vision-language models to generate accurate and clinically relevant radiology reports from chest X-ray images. It probes both linguistic fluency and coverage against reference reports, as well as clinical accuracy in identifying pathological findings. Use when the user wants to benchmark on IU X-ray, MIMIC-CXR, CheXpert Plus, or asks about evaluating this task. Reports ROUGE-L.

Evaluates an LLM's ability to perform cross-lingual retrieval-augmented generation by answering questions in a target language using supporting documents in English or mixed languages, while ignoring topically related distractors. It specifically probes cross-document reasoning capabilities and response language consistency. Use when the user wants to benchmark on XRAG, or asks about evaluating this task. Reports response language consistency.

Probes large-scale 3D scene understanding by evaluating a model's ability to reason across multiple rooms, locate specific objects, generate embodied task plans, and produce detailed captions in complex, high-density environments. It specifically tests spatial awareness, contextual inference, and fine-grained detail retention beyond single-room benchmarks. Use when the user wants to benchmark on XR-Scene, or asks about evaluating this task. Reports CIDEr.

Evaluates automatic machine translation metrics by measuring their correlation with human quality judgments across multiple language pairs. It probes whether metrics exhibit cross-lingual scoring bias and how reliably they rank translation systems or quality triplets relative to human assessments. Use when the user wants to benchmark on XQ-MEval, or asks about evaluating this task. Reports Kendall-τ.

This benchmark probes the cross-modal consistency and reasoning capabilities of omni-language models by evaluating semantic equivalence across all six possible modality combinations (text, vision, audio) for both context and candidate inputs. It measures how well models maintain performance when modalities are swapped or combined, highlighting modality-specific biases and directional asymmetries. Use when the user wants to benchmark on XModBench, or asks about evaluating this task. Reports ac...

This benchmark evaluates the cross-lingual transfer capability of neural news recommenders. It probes how well models trained monolingually on English news can generate accurate recommendations in 14 other languages under zero-shot and few-shot settings, with and without bilingual user consumption patterns. Use when the user wants to benchmark on xMIND, or asks about evaluating this task. Reports AUC.

Evaluates multimodal large language models on ultra-high-resolution remote sensing imagery using vision-language question answering. It probes both perception (e.g., object classification, counting, spatial relations) and reasoning capabilities across various sub-tasks. Use when the user wants to benchmark on XLRS-Bench, LRS-VQA, or asks about evaluating this task. Reports accuracy.

Evaluates the ability of text-to-SQL models to generate correct SQL or GQL queries for natural language questions across relational and graph databases. It measures execution accuracy by comparing the runtime results of generated queries against reference queries on specific database instances. Use when the user wants to benchmark on Spider, Bird, SQL-Eval, NL2GQL, or asks about evaluating this task. Reports Execution Accuracy (EX).

Evaluates a Text-to-SQL framework's ability to generate correct and efficient SQL queries from natural language questions across complex, cross-domain databases. It probes schema filtering, multi-generator candidate creation, and selection robustness. Use when the user wants to benchmark on BIRD, Spider, or asks about evaluating this task. Reports Execution Accuracy (EX).

Evaluates cross-lingual transfer capabilities of pre-trained language models across 11 diverse natural language understanding and generation tasks spanning over 100 languages. It measures how well models fine-tuned on English can generalize to zero-shot testing in other languages. Use when the user wants to benchmark on XGLUE, or asks about evaluating this task. Reports accuracy.

Evaluates the ability of graph neural networks to detect fraudulent transactions in large-scale, highly imbalanced e-commerce transaction graphs. It probes model performance under extreme class imbalance and measures the trade-off between detection accuracy, inference speed, and scalability across different graph sizes and distributed settings. Use when the user wants to benchmark on eBay-xlarge, eBay-large, eBay-small, or asks about evaluating this task. Reports AUC.

Evaluates a model's ability to detect violent events in untrimmed audio-visual videos under weak supervision. It probes the model's capacity to fuse complementary audio and visual cues to localize violence in class-imbalanced, long-range video sequences. Use when the user wants to benchmark on XD-Violence, or asks about evaluating this task. Reports AP.

Evaluates the ability of multi-agent LLM frameworks to automatically fix buggy Ruby code using test-driven feedback loops. It probes iterative code repair, self-reflection, and test generation capabilities under varying difficulty levels and error types. Use when the user wants to benchmark on xCodeEval (Ruby subset), or asks about evaluating this task. Reports pass@1.

Evaluates large language models on multilingual code understanding, generation, translation, and retrieval across 11 programming languages. It probes the model's ability to produce executable, correct code by validating outputs against unit tests rather than relying on lexical overlap. Use when the user wants to benchmark on xCodeEval, or asks about evaluating this task. Reports pass@5.

Evaluates a model's ability to perform cross-cultural machine translation, specifically probing its capacity to accurately transcreate culturally nuanced entity names across multiple language pairs rather than merely transliterating or omitting them. Use when the user wants to benchmark on XC-Translate, WMT (17-21), or asks about evaluating this task. Reports M-ETA.

Evaluates a machine learning model's ability to predict local chemical descriptors (oxidation state and coordination number) from experimental X-ray absorption spectra, specifically testing how well spectral domain mapping bridges the gap between simulated training data and real experimental measurements. Use when the user wants to benchmark on Combinatorial Zinc Titanate Thin Film XANES, or asks about evaluating this task. Reports OS/CN prediction accuracy.

Evaluates the reliability and operational suitability of white-box explainable AI methods (DeepLift, Integrated Gradients, LRP) when applied to deep neural network-based intrusion detection systems. It probes how well these methods preserve model accuracy, maintain consistency under repeated runs, resist adversarial noise, and compute efficiently across real-world network traffic datasets. Use when the user wants to benchmark on NSL-KDD, RoEduNet-SIMARGL2021, CICIDS-2017, or asks about evalua...

Evaluates the computational efficiency and explanation fidelity of an ROI-guided perturbation framework for medical image segmentation. It measures how effectively the method reduces computation while preserving segmentation accuracy and explanation quality compared to full occlusion baselines. Use when the user wants to benchmark on FLARE22, SAROS, CHAOS, or asks about evaluating this task. Reports Dice coefficient.