Research

Evaluates an integer linear programming model for allocating network monitoring depth across protocol layers, balancing detection efficiency against computational resource constraints on a synthetic heterogeneous network. Use when the user wants to benchmark on Synthetic 6-Device Network, or asks about evaluating this task. Reports objective function value.

Evaluates a machine learning model's ability to predict the probability of hydrocarbon reservoir presence in a 3D geological space using seismic and well log data. It probes the model's capacity for binary lithological classification and probabilistic calibration under early-stage exploration conditions with limited well data. Use when the user wants to benchmark on Achimov sedimentary complex field dataset, or asks about evaluating this task. Reports classification quality.

Evaluates a model's ability to perform N-ary relation extraction from scientific documents by first retrieving relevant text/table components and then selecting the correct entities within those components. Use when the user wants to benchmark on SciREX, PubMed, NLP-TDMS (Full), or asks about evaluating this task. Reports Accuracy (Acc).

Evaluates whether a multi-agent research simulator can accurately reconstruct masked research nodes (papers and reviews) from their local neighborhood context in a collaborative graph. It probes the model's ability to capture interdisciplinary collaboration patterns and realistic academic writing styles. Use when the user wants to benchmark on ResearchTown simulated community graph, or asks about evaluating this task. Reports reconstruction_similarity.

Evaluates the capability of LLM agents to conduct closed-loop scientific research by proposing hypotheses, executing experiments, and outperforming human baselines on repurposed real-world AI papers. It probes long-horizon planning, resource management, and autonomous experimentation under realistic tool constraints. Use when the user wants to benchmark on ResearchGym, or asks about evaluating this task. Reports improvement over baselines.

Evaluates the ability of LLMs and AI agents to automatically repair broken R-based social science code and restore computational reproducibility. It probes how well different workflows handle varying error complexities and contextual information. Use when the user wants to benchmark on Custom R-based Social Science Code Dataset, or asks about evaluating this task. Reports reproduction_success_rate.

Evaluates whether agentic AI systems can accurately assess the computational reproducibility of social science research by comparing original paper findings against results reproduced from provided raw data and code. It probes end-to-end agentic reasoning, including command execution, debugging, and result interpretation in a simulated research environment. Use when the user wants to benchmark on REPRO-Bench, or asks about evaluating this task. Reports accuracy.

This benchmark evaluates repository-level code completion by measuring how accurately a model predicts missing code segments given surrounding context and retrieved repository snippets. It probes both syntactic similarity and functional correctness across line, API, and function-level granularity. Use when the user wants to benchmark on RepoEval, or asks about evaluating this task. Reports Exact Match (EM).

Evaluates a model's ability to retrieve relevant cross-file code snippets given an in-file context for predicting the next line of code. Use when the user wants to benchmark on RepoBench-R, or asks about evaluating this task. Reports acc@1.

Evaluates an end-to-end pipeline that first retrieves cross-file snippets and then predicts the next line of code using both the in-file context and retrieved snippets. Use when the user wants to benchmark on RepoBench-P, or asks about evaluating this task. Reports Exact Match (EM).

Evaluates autoregressive language models on predicting the next line of code using provided in-file and cross-file contexts. Use when the user wants to benchmark on RepoBench-C, or asks about evaluating this task. Reports Exact Match (EM).

Evaluates LLMs' ability to read unseen reference documents and answer questions based solely on the provided context, as well as their ability to detect unanswerable questions and classify document topics. It specifically probes whether models rely on pre-training memory versus actual context-conditional reading and retrieval skills. Use when the user wants to benchmark on RepLiQA, TriviaQA, or asks about evaluating this task. Reports recall.

Evaluates the robustness of audio deepfake detection models against replay attacks where deepfake audio is played back and re-recorded through real-world hardware, introducing acoustic distortions and room impulse responses. Use when the user wants to benchmark on ReplayDF, or asks about evaluating this task. Reports EER (%).

Evaluates session-based recommendation models on predicting the next item in a user session. It probes the model's ability to capture sequential patterns and handle repeat consumption behaviors across e-commerce and music domains. Use when the user wants to benchmark on YOOCHOOSE, DIGINETICA, LASTFM, or asks about evaluating this task. Reports Recall@k.

Probes the robustness and cross-condition consistency of multimodal large language models on Earth observation tasks under realistic visual and textual perturbations. Evaluates performance degradation and behavioral stability across clean and perturbed inputs for scene classification, VQA, and visual grounding. Use when the user wants to benchmark on RemoteShield clean-perturbed benchmarks, or asks about evaluating this task. Reports RPD, CCA.

Evaluates in-domain representation learning and scene classification across diverse remote sensing modalities (optical, SAR, aerial) and spatial resolutions. It probes model robustness to varying class balances, visual similarities, and label domains in Earth observation. Use when the user wants to benchmark on BigEarthNet, EuroSAT, RESISC-45, So2Sat, UC Merced, or asks about evaluating this task. Reports accuracy.

Evaluates vision-language models' ability to perform multi-view spatial reasoning, including relative direction, relative distance, cross-view consistency, and perspective-taking. It probes whether models can maintain geometric coherence and integrate information across multiple camera viewpoints to answer questions about procedurally generated indoor scenes. Use when the user wants to benchmark on ReMindView-Bench, or asks about evaluating this task. Reports accuracy.

Evaluates whether pointwise re-rankers can function as binary relevance judges by predicting whether a document is relevant to a query. It probes the capability of adapted ranking models to perform direct relevance classification and compares their performance against LLM-based judges. Use when the user wants to benchmark on TREC-DL, or asks about evaluating this task. Reports binary accuracy.

Evaluates the ability of graph neural networks to learn from relational databases by predicting entity attributes (classification and regression) and discovering relationships between entities (link prediction) using primary-foreign key graph structures. Use when the user wants to benchmark on RELBENCH, or asks about evaluating this task. Reports AUROC.

Evaluates a model-agnostic fairness calibration heuristic that adjusts prediction thresholds per protected attribute value to independently control false positive and false negative rates. It probes the ability to balance business-critical error costs across high-arity and multiple sensitive groups while maintaining predictive performance. Use when the user wants to benchmark on COMPAS (Criminal Recidivism), Income-Prediction, Health Prediction, Proprietary Online Fraud Dataset, or asks about...

Evaluates the precision of estimating globular cluster distance and mass parameters by quantifying how much gravitational wave modulation reduces parameter uncertainties compared to electromagnetic-only observations. Use when the user has predictions and gold and needs to compute relative_uncertainty.

Evaluates the prediction accuracy of data-driven neural network closures for kinetic theory of active fluids against reference kinetic simulations and traditional analytical closures. It probes the model's ability to capture rotational symmetries, generalize across parameter regimes, and resolve topological defects in fluid dynamics. Use when the user has predictions and gold and needs to compute RMSE.

Evaluates the effectiveness of machine-learned preconditioners for linear solvers in semi-implicit shallow-water models by measuring error reduction in the first solver iteration and overall convergence rates during free-running simulations. Use when the user has predictions and gold and needs to compute Relative decrease in MAE.

This benchmark probes an LLM's ability to perform high-arity relational reasoning and multi-constraint integration across scientific domains. It isolates the difficulty of jointly binding independent entities to satisfy a relation, independent of prompt length or in-context learning. Use when the user wants to benchmark on REL, or asks about evaluating this task. Reports accuracy.