Research

Measures how well automated RAG scoring metrics align with human preferences in pairwise comparison tasks. It probes the ability of reference-free faithfulness, answer relevance, and context relevance estimators to replicate human judgment on answer and context quality. Use when the user wants to benchmark on WikiEval, or asks about evaluating this task. Reports accuracy.

Evaluates knowledge graph models on node and entity classification under edge incompleteness, and link prediction with varying feature/class configurations. It probes how well models leverage relational structure, node features, and joint link prediction capacity to handle missing edges and predict missing links. Use when the user wants to benchmark on WikiAlumni, or asks about evaluating this task. Reports validation accuracy.

Evaluates end-to-end speech-to-speech (S2S) language models on real-world conversational tasks, probing their ability to handle diverse query types, paralinguistic features (prosody, disfluencies), and robustness to background noise. Use when the user wants to benchmark on WildSpeech-Bench, or asks about evaluating this task. Reports Score.

Evaluates face detection algorithms on real-world images with extreme variations in scale, pose, occlusion, and event context. It probes the ability of detectors to handle small faces, heavy occlusion, and atypical poses under standard bounding box matching criteria. Use when the user wants to benchmark on WIDER FACE, or asks about evaluating this task. Reports Average Precision (AP).

Evaluates a model's ability to perform broad information seeking by decomposing complex queries into parallel subtasks and producing structured tabular outputs. It also measures robustness on standard single-hop and multi-hop open-domain QA tasks. Use when the user wants to benchmark on WideSearch, or asks about evaluating this task. Reports item F1 score.

Evaluates a hybrid recommender model's ability to balance memorization of frequent user-item interactions with generalization to unseen combinations for ranking candidate apps. The protocol measures predictive accuracy on a static holdout set and business impact via live A/B testing on user acquisition rates. Use when the user wants to benchmark on Google Play App Store (internal), or asks about evaluating this task. Reports Online Acquisition Gain.

Evaluates the zero-shot generalization capability of a speech recognition model across diverse English and multilingual domains. It measures robustness to out-of-distribution audio, varying noise levels, and translation tasks without any dataset-specific fine-tuning. Use when the user wants to benchmark on LibriSpeech, Common Voice, Fleurs, CoVoST2, Multilingual LibriSpeech (MLS), VoxPopuli, or asks about evaluating this task. Reports WER.

Evaluates single-channel speech separation and enhancement (denoising/dereverberation) capabilities under realistic noisy and reverberant conditions using synthetically generated reverberant mixtures. Use when the user wants to benchmark on WHAMR!, or asks about evaluating this task. Reports SI-SDR.

Evaluates neural language models' ability to understand Words of Estimative Probability (WEP) by testing their capacity to distinguish valid from invalid probabilistic verbalizations and perform logical consistency checks in probabilistic reasoning. Use when the user wants to benchmark on WEP Reasoning 1 hop, WEP Reasoning 2 hops, WEP-UNLI, or asks about evaluating this task. Reports accuracy.

Evaluates speech processing capabilities for the Chinese Wu dialect, including automatic speech recognition (ASR), automatic speech translation (AST), speaker attribute prediction (gender, age), emotion recognition, text-to-speech (TTS), and instruction-following TTS. Use when the user wants to benchmark on WenetSpeech-Wu-Bench, or asks about evaluating this task. Reports CER (%).

Evaluates Mandarin speech recognition systems across diverse, real-world domains (internet, meetings) and mixed Mandarin-English content. It benchmarks the robustness of ASR models against noisy, production-level audio and varying data scales. Use when the user wants to benchmark on WenetSpeech, or asks about evaluating this task. Reports MER%.

Evaluates a model's ability to jointly perform fine-grained weed species segmentation, continuous plant height regression, and discrete temporal growth stage classification from single RGB images. Use when the user wants to benchmark on WeedSense, or asks about evaluating this task. Reports mIoU, MAE, Accuracy.

Evaluates LLM-based agents' ability to systematically navigate multi-layered, real-world websites to extract buried information. It probes long-range reasoning, memory management, and step-by-step click-based navigation under strict action limits. Use when the user wants to benchmark on WebWalkerQA, or asks about evaluating this task. Reports accuracy.

Measures an autonomous web agent's end-to-end task completion capability across dynamic, real-world websites. It evaluates multi-step navigation, form filling, and robustness in open-web environments. Use when the user wants to benchmark on WebVoyager, or asks about evaluating this task. Reports Success Rate.

Evaluates the robustness and accuracy of deep object trackers in challenging underwater environments. It probes cross-domain adaptation from open-air to underwater domains, as well as within-domain fine-tuning capabilities, while also assessing performance under varying frame rates and complex visual conditions like occlusion and low visibility. Use when the user wants to benchmark on WebUOT-1M, or asks about evaluating this task. Reports AUC.

Evaluates the robustness and accuracy of deep visual tracking algorithms on large-scale, real-world UAV video sequences. It probes how well trackers handle diverse environmental conditions, motion dynamics, and target appearance changes without parameter tuning. Use when the user wants to benchmark on WebUAV-3M, or asks about evaluating this task. Reports AUC.

Evaluates a model's ability to extract main content from HTML web pages by classifying semantic blocks and generating clean text or Markdown. It probes robustness across varying difficulty levels and rich content types such as tables, code, and equations. Use when the user wants to benchmark on WebMainBench, WCEB, or asks about evaluating this task. Reports ROUGE-N F1.

This benchmark evaluates visual web agents on their ability to navigate complex, multi-step tasks across diverse websites and domains. It specifically probes long-horizon interaction, information extraction, and out-of-distribution generalization to unseen websites. Use when the user wants to benchmark on WebGym, or asks about evaluating this task. Reports accuracy.

Evaluates a model's ability to generate functional and visually accurate website codebases from natural language instructions. It measures both functional correctness via automated GUI-agent testing and visual fidelity via VLM-based appearance scoring. Use when the user wants to benchmark on WebGen-Bench, or asks about evaluating this task. Reports Accuracy.

Evaluates browser agents' ability to complete interactive web tasks across varying difficulty levels and domains. It probes navigation, visual understanding, multi-step reasoning, and interaction capabilities in realistic, self-contained web environments. Use when the user wants to benchmark on WebForge-Bench, or asks about evaluating this task. Reports accuracy (%).

Evaluates fine-grained image classification performance on large-scale webly supervised datasets characterized by label noise and extreme class imbalance. It probes a model's ability to learn robust visual features and correct noisy labels through mutual peer learning or standard fine-tuning. Use when the user wants to benchmark on WebFG-496, WebiNat-5089, or asks about evaluating this task. Reports classification accuracy (%).

Evaluates multilingual dense retrieval models on natural Q&A pairs by measuring ranking quality against gold answers. It tests the model's ability to retrieve relevant FAQ documents across multiple languages and assesses zero-shot generalization to other Wikipedia-based benchmarks. Use when the user wants to benchmark on WebFAQ, Mr. TyDi, MIRACL (Hard Negatives), or asks about evaluating this task. Reports NDCG@10.

Evaluates LLMs' ability to generate complete web applications from real-world user requirements. It probes multi-modal understanding, code generation quality, and strict adherence to ground-truth checklists across functionality, visual design, and content dimensions. Use when the user wants to benchmark on WebCoderBench, or asks about evaluating this task. Reports checklist-based evaluation.

This benchmark evaluates multimodal models' ability to translate webpage design screenshots into functional HTML/CSS code. It probes visual fidelity, structural hierarchy recall, and the capacity to generate long, complex, real-world front-end code from visual inputs. Use when the user wants to benchmark on WebCode2M, or asks about evaluating this task. Reports TreeBLEU.