Research

Evaluates network intrusion detection capability by classifying network traffic flows as benign or malicious (or specific attack types) using graph-structured representations of network connections. It probes the model's ability to learn from adaptive graph construction and contrastive learning under resource-constrained conditions. Use when the user wants to benchmark on UNSW-NB15, or asks about evaluating this task. Reports accuracy.

Evaluates the ability of language models to perform unsupervised relation extraction by predicting relation labels or tokens from contextual text. It probes factual grounding and context-constrained generation capabilities across varying relation types and corpus sources. Use when the user wants to benchmark on T-REx, Google-RE, ZSRE, TACRED, or asks about evaluating this task. Reports F1.

Evaluates the ability of image descriptors to distinguish near-duplicate image pairs from non-duplicates under extreme specificity constraints, simulating large-scale forensic or fraud detection scenarios. Use when the user wants to benchmark on MFND (Mir-Flickr Near-Duplicate), CLAIMS, Holidays, California-ND, or asks about evaluating this task. Reports sensitivity at false positive rate (FPR).

This benchmark evaluates a model's ability to detect unsupervised lexical semantic change across diachronic corpus pairs. It probes two capabilities: binary classification of whether a word's sense has been gained or lost, and ranking the intensity of semantic change relative to a gold standard. Use when the user wants to benchmark on SemEval-2020 Task 1, or asks about evaluating this task. Reports accuracy, Spearman’s rank-order correlation coefficient.

Evaluates whether language models exhibit gender bias when processing sentence pairs that have been filtered to remove explicit gendered language and stereotypical co-occurrences. It measures the model's ability to generate gender-neutral completions and checks for systematic preference toward male or female pronouns in stereotype-free contexts. Use when the user wants to benchmark on USE-5, USE-10, USE-20, WB (Winobias), WG (Winogender), or asks about evaluating this task. Reports US fairnes...

Evaluates a model's ability to recognize emotions in speech from speakers it has never encountered during training. It probes cross-speaker generalization and robustness to acoustic variability across multiple languages and recording conditions. Use when the user wants to benchmark on CREMA-D, IEMOCAP, RAVDESS, EmoDB, CaFE, BhavVani, or asks about evaluating this task. Reports WF1.

Evaluates a model's ability to estimate the 6D pose (rotation and translation) of novel, unseen 3D objects in real-world scenes without retraining, using only their mesh models and partial RGBD inputs. It specifically probes robustness to pose ambiguity, partial observability, and real-world noise. Use when the user wants to benchmark on GraspNet-1Billion, YCB-Video, or asks about evaluating this task. Reports IADD.

Evaluates the effectiveness of image safety classifiers in detecting various unsafe content categories across real-world and AI-generated images. It also probes classifier robustness to distribution shifts caused by artistic representations and grid layouts in AI-generated content. Use when the user wants to benchmark on UnsafeBench, or asks about evaluating this task. Reports F1-Score.

Evaluates embodied AI agents' capabilities in complex, photo-realistic 3D open-world environments. Specifically probes visual navigation on unstructured terrain, active visual tracking across diverse scenes, and social tracking under dynamic distractions, varying morphologies, and different control frequencies. Use when the user wants to benchmark on UnrealZoo, or asks about evaluating this task. Reports Success Rate (SR).

Assesses multimodal language models' ability to resolve lexical ambiguity in puns using visual context. It probes visual-textual alignment, multimodal literacy, and the capacity to disambiguate or reconstruct ambiguous text when provided with explanatory or disambiguating images. Use when the user wants to benchmark on UNPIE, or asks about evaluating this task. Reports exact-match accuracy.

Evaluates the ability of recommender systems to efficiently remove specific user interactions or sensitive items (unlearning) while preserving recommendation utility. It probes real-world operational constraints, including handling sequential small-batch deletion requests, domain-specific triggers, and low-latency execution across collaborative filtering, session-based, and next-basket recommendation tasks. Use when the user wants to benchmark on TaFeng, Dunnhumby, Instacart, RSC15, DIGI, NOW...

Evaluates a model's ability to recognize human actions from heterogeneous skeleton data with varying joint counts and topologies. It probes cross-domain generalization, zero-shot/few-shot transfer, and robustness to structural discrepancies between sensing modalities. Use when the user wants to benchmark on NTU-60, HumanML3D, NW-UCLA, NTU-120, or asks about evaluating this task. Reports Accuracy.

Evaluates multilingual named entity recognition (NER) capabilities across 22 languages and 30 datasets, probing both in-language performance and cross-lingual transfer. It also benchmarks large language models as annotators against human inter-annotator agreement to assess guideline adherence and annotation quality. Use when the user wants to benchmark on UNER v2, or asks about evaluating this task. Reports micro F1.

This benchmark probes an LLM agent's ability to perform spatial-temporal reasoning and generate executable code for Earth Observation tasks. It evaluates whether models can correctly answer yes/no questions derived from scientific articles by leveraging remote sensing data via Google Earth Engine. Use when the user wants to benchmark on UnivEARTH, or asks about evaluating this task. Reports accuracy.

Evaluates a video agent's capabilities across generation, understanding, editing, and segmentation tasks, while probing its agentic planning and memory mechanisms. It measures how well a unified agent architecture handles long-horizon, multi-step video workflows compared to monolithic baselines. Use when the user wants to benchmark on UniVA-Bench, or asks about evaluating this task. Reports MLLM Judge.

This evaluation measures the transcription accuracy of a fine-tuned automatic speech recognition model across four diverse speech benchmarks. It specifically probes the model's robustness to different speaking styles, accents, and linguistic contexts after applying a noise reduction step and a BART-based semantic correction pipeline. Use when the user wants to benchmark on LibriSpeech, Europarl-ASR, TED-LIUM, FLEURS, or asks about evaluating this task. Reports Word Error Rate (WER).

Evaluates text-to-SQL models on compositional generalization, out-of-domain robustness, and schema-question alignment across 18 diverse datasets and 12 domains. It probes the model's ability to handle long-form query decomposition and cross-domain SQL pattern diversity. Use when the user wants to benchmark on UNITE, or asks about evaluating this task. Reports accuracy.

Evaluates text summarization models across multiple dimensions including faithfulness, completeness, conciseness, domain stability, and abstractiveness. It tests how well summarizers handle diverse input contexts (domains, dialogue vs. non-dialogue, short vs. long texts) and the impact of PII redaction on hallucination. Use when the user wants to benchmark on UniSumEval, or asks about evaluating this task. Reports faithfulness.

Evaluates multimodal voice generation and conversion capabilities across face-driven, text-driven, and attribute-based tasks. Probes the model's ability to align facial, textual, and attribute descriptions with target speech while preserving speaker identity, content clarity, and naturalness. Use when the user wants to benchmark on LRS3, or asks about evaluating this task. Reports MOS-Match.

Evaluates a unified text-to-audio model's ability to generate speech, music, and sound effects from natural language instructions without reference audio. It probes instruction-following fidelity, acoustic quality, structural coherence, and the positive transfer effects of multi-modal joint training. Use when the user wants to benchmark on UniSonate Unified Corpus, Seed-TTS test set, SongEval benchmark, or asks about evaluating this task. Reports WER, SongEval.

Evaluates multimodal large language models on perceptual-level image understanding across three domains: Image Aesthetics & Art (IAA), Image Quality Assessment (IQA), and Image Structure & Texture Assessment (ISTA). It probes both continuous visual rating (VR) and discrete visual question answering (VQA) capabilities. Use when the user wants to benchmark on UniPercept-Bench, or asks about evaluating this task. Reports Acc..

Evaluates a unified discrete diffusion framework's capability to jointly generate and reason over image-text pairs. It probes unconditional and conditional generation quality, the effectiveness of classifier-free guidance, training and inference efficiency, and cross-modal retrieval and reasoning performance. Use when the user wants to benchmark on DataComp1B, CC12M, MS-COCO30k, Flickr, Winoground, or asks about evaluating this task. Reports FID.

Evaluates large language models' factual correctness by dynamically generating responses to factual questions and measuring how well hallucination detection and fact verification methods can predict the ground-truth factuality label of those responses. It probes a model's susceptibility to hallucination and the effectiveness of external evidence retrieval in verifying generated claims. Use when the user wants to benchmark on TriviaQA, NQ-Open, PopQA, 2WikiMultihopQA, HotpotQA, or asks about e...

Evaluates natural language generation models across multiple quality dimensions (e.g., coherence, fluency, consistency, relevance) by reframing assessment as a Boolean QA task. Measures how well automated scores align with human judgments using correlation metrics. Use when the user wants to benchmark on SummEval, Topical-Chat, SFRES, SFHOT, QAGS, or asks about evaluating this task. Reports Spearman correlation.