Research

Evaluates a model's ability to automatically generate concise, coherent language summaries of mobile UI screens by fusing visual, structural, and textual modalities. It probes multimodal representation learning and language generation capabilities in the context of human-computer interaction and UI understanding. Use when the user wants to benchmark on Screen2Words, or asks about evaluating this task. Reports BLEU-4.

Evaluates a model's ability to locate specific GUI elements from a screenshot given a text instruction. It measures both coarse localization accuracy and fine-grained bounding box overlap across desktop, mobile, and web platforms. Use when the user wants to benchmark on ScreenSpot, or asks about evaluating this task. Reports grounding accuracy.

This evaluation probes a vision-language model's ability to localize specific UI elements within graphical user interfaces based on natural language instructions. It tests precise coordinate prediction and cross-resolution generalization across mobile, desktop, and web platforms. Use when the user wants to benchmark on ScreenSpot, ScreenSpot-v2, ScreenSpot-Pro, or asks about evaluating this task. Reports accuracy.

Evaluates a model's ability to detect, localize, and semantically label all interactable UI elements on a clean screenshot. It probes fine-grained spatial reasoning, handling of dense layouts, and UI semantics understanding. Use when the user wants to benchmark on GUI-360°-Bench, or asks about evaluating this task. Reports F1.

Evaluates LLMs' ability to understand, diagnose, and repair bugs in multimodal, event-driven block-based programming environments (Scratch). It probes functional correctness, structured bug explanation, trigger/mechanism identification, and patch minimality/semantic preservation. Use when the user wants to benchmark on ScratchEval, or asks about evaluating this task. Reports G-Acc.

Evaluates LLM-based web information extraction by measuring structural validity (JSON parseability, schema compliance), key extraction accuracy (precision, recall, F1), and value extraction quality (type-aware exact match, BLEU) on real-world HTML-to-JSON tasks. Use when the user wants to benchmark on ScrapeGraphAI-100k, or asks about evaluating this task. Reports Key F1.

Evaluates the scientific reasoning and problem-solving capabilities of LLMs on graduate-level higher education science problems. It measures how well models can parse and solve complex scientific questions involving formulas and equations. Use when the user wants to benchmark on SCP-116K, or asks about evaluating this task. Reports Accuracy.

This framework audits medical LLM benchmarks across five lifecycle phases using 46 medically tailored criteria to assess clinical relevance, data integrity, safety-critical capabilities, validity, and governance. Use when the user has predictions and gold and needs to compute score.

This evaluation protocol measures the impact of score ties on document ranking repeatability across diverse information retrieval collections. It quantifies how non-deterministic tie-breaking during multi-threaded indexing causes variability in standard ranking metrics, even when using identical queries and ranking models. Use when the user wants to benchmark on TREC 2004 Robust Track (Disks 4 & 5), TREC 2005 Robust Track (AQUAINT), TREC 2017 Common Core Track (NYT Annotated Corpus), TREC 201...

Evaluates dense optical flow estimation accuracy and occlusion detection on standard video benchmarks. Probes the model's ability to predict pixel-wise motion vectors and identify occluded regions under varying motion magnitudes and scene complexities. Use when the user wants to benchmark on Sintel, KITTI, or asks about evaluating this task. Reports End Point Error (EPE).

This evaluation protocol assesses the ability of deep learning models to predict drug-target interactions (DTI) by learning from molecular graphs and protein sequences. It probes the model's capacity to capture cross-domain interaction patterns between small molecules and proteins, particularly in semi-inductive settings where novel compounds are paired with known protein families. Use when the user wants to benchmark on BindingDB, KIBA, Human, SCOPE, or asks about evaluating this task. Repor...

Evaluates computational methods for single-cell multi-omics integration by measuring their ability to preserve biological variation, align different omics layers at the cell and single-cell levels, and enable accurate cell type annotation. Use when the user wants to benchmark on SHARE-seq BMMC, SNARE-seq, 10X Genomics Multiome, Human fetal atlas, CITE-seq BMMC S1, CITE-seq BMMC S4, Human brain multi-omics, Human brain 3k, or asks about evaluating this task. Reports biological variation conser...

Evaluates the fidelity of a particle filter algorithm for generating counterfactual samples from structural causal models by comparing empirical statistics of the generated samples against known ground-truth distributions and correlations. Use when the user wants to benchmark on Synthetic SCM Simulation, or asks about evaluating this task. Reports proportion of unique observations.

Evaluates a model's ability to perform hierarchical scientific summarization by generating three distinct granularity levels (Abstract, Key Contributions, TL;DR) from a single full-text input. It probes multi-granularity text compression and the model's capacity to maintain coherence across varying compression ratios within a single inference pass. Use when the user wants to benchmark on SciZoom, or asks about evaluating this task. Reports unspecified summarization metric.

Evaluates multimodal LLMs on closed-ended visual and non-visual question answering over scientific figures. It probes recognition of visual attributes (color, shape, position) and reasoning capabilities across diverse chart types. Use when the user wants to benchmark on SciVQA, or asks about evaluating this task. Reports ROUGE-1 F1.

Evaluates large language models across four dimensions of trustworthiness in scientific contexts: truthfulness, adversarial robustness, scientific safety, and scientific ethics. It probes models' ability to provide accurate scientific information, resist adversarial perturbations, avoid generating harmful content, and make sound ethical judgments in research scenarios. Use when the user wants to benchmark on SciQ, ARC-C, MMLU, GPQA-Diamond, LogiQA, ReClor, LOGICINFERENCE, WMDP, HarmBench, Sci...

Evaluates long-context language models' ability to perform numerical aggregation, filtering, sorting, and logical operations across extended contexts (up to 1M tokens) using scientific article metadata and full-text articles. Use when the user wants to benchmark on SciTrek, or asks about evaluating this task. Reports exact match.

This benchmark evaluates the ability of models to generate extreme, single-sentence summaries (TLDRs) of scientific papers, capturing key contributions while bypassing background details. It tests both automated overlap metrics and human-judged informativeness and correctness under multi-target and multi-input settings. Use when the user wants to benchmark on SCITLDR, or asks about evaluating this task. Reports Rouge-1.

Evaluates an LLM's ability to comprehend algorithmic descriptions from academic papers and translate them into executable code. It probes the model's capacity for algorithmic reasoning, dependency resolution, and practical implementation within a repository context. Use when the user wants to benchmark on SciReplicate-Bench, or asks about evaluating this task. Reports Execution Accuracy.

Evaluates open-ended, closed-book scientific question answering capabilities. It probes a model's ability to generate comprehensive, accurate, and reasonable answers to research-level science questions without external context or reference papers. Use when the user wants to benchmark on SciQAG-24D, SciQ, or asks about evaluating this task. Reports CAR.

Evaluates a model's ability to perform complex, claim-centric reasoning over full scientific documents containing multimodal elements (charts, tables, figures). It probes the model's capacity to localize evidence and answer questions accurately despite long-context noise and distractors. Use when the user wants to benchmark on SciMDR-Eval, or asks about evaluating this task. Reports accuracy.

Probes large language models' scientific knowledge across five progressive cognitive levels: memory, comprehension, reasoning, ethical discernment, and real-world application. Covers four scientific domains (biology, chemistry, physics, materials science) using diverse question formats including multiple-choice, relation extraction, and open-ended protocol design. Use when the user wants to benchmark on SciKnowEval, or asks about evaluating this task. Reports overall normalized score.

Evaluates multi-modal large language models' ability to interpret scientific graphs and generate accurate, context-aware answers in a multi-turn conversational setting. It probes open-vocabulary visual reasoning and the model's capacity to leverage auxiliary paper metadata for grounded responses. Use when the user wants to benchmark on SciGraphQA, or asks about evaluating this task. Reports CIDEr.

Evaluates the logical correctness, structural fidelity, and information utility of AI-generated scientific images. It probes whether generated visuals accurately encode domain-specific facts and geometric relationships, and whether they are indispensable for solving visually grounded scientific quizzes. Use when the user wants to benchmark on SciGenBench, or asks about evaluating this task. Reports inverse_validation_rate.