Research

Evaluates Chinese large language models on five biomedical NLP tasks: information extraction, text classification, natural language inference, dialogue understanding, and content generation. It probes instruction-following, few-shot in-context learning, and parameter-efficient fine-tuning capabilities in a medical domain context. Use when the user wants to benchmark on PromptCBLUE, or asks about evaluating this task. Reports Instance-level strict micro-F1.

This benchmark evaluates the vulnerability of large language models to prompt injection attacks when generating scientific paper reviews. It probes whether hidden or biased instructions embedded in parsed PDFs can systematically skew the model's review scores and recommendations. Use when the user wants to benchmark on ICLR 2024 Review Dataset, or asks about evaluating this task. Reports Rating.

This benchmark evaluates an LLM's or monitoring system's ability to distinguish between safe user inputs and malicious prompt injection attacks. It measures both false positive rates on legitimate interactions and false negative rates on adversarial prompts to assess overall security robustness. Use when the user wants to benchmark on Gandalf, Tensor-Trust, SPML-Dataset, or asks about evaluating this task. Reports Error Rate (ER).

Evaluates 3D volumetric medical image segmentation capability on prostate MRI scans. It probes the model's ability to accurately delineate organ boundaries under clinical variability and class imbalance using end-to-end fully convolutional networks. Use when the user wants to benchmark on PROMISE2012, or asks about evaluating this task. Reports Dice coefficient.

Evaluates the fine-grained judgment capability of vision-language models by scoring generated text outputs against instance-specific rubrics and reference answers. It measures alignment with human preferences and state-of-the-art VLM judges across instruction following, VQA, and captioning tasks. Use when the user wants to benchmark on LLaVA-Bench, VisIT-Bench, Perception-Bench, OKVQA, VQAv2, TextVQA, COCO-Captions, NoCaps, or asks about evaluating this task. Reports Pearson correlation.

This evaluation protocol assesses the capability of language models to act as automated judges for other language models. It probes two distinct paradigms: direct assessment, where a model scores a single response against a reference or rubric, and pairwise ranking, where a model selects the preferred response between two candidates. The benchmarks cover instruction-following, alignment, and fine-grained custom criteria. Use when the user wants to benchmark on Vicuna Bench, MT Bench, FLASK, F...

Evaluates how well discovered motif sets in time series approximate ground truth motif sets. It penalizes false positives, false negatives, and redundant motifs without requiring uniform motif lengths or a fixed number of motif sets. Use when the user has predictions and gold and needs to compute PROM.

Evaluates a progressive feature transmission protocol for split inference at the wireless edge, measuring how efficiently features are transmitted to meet target inference accuracy or uncertainty thresholds under varying channel conditions. Use when the user wants to benchmark on GM dataset, MNIST, or asks about evaluating this task. Reports average communication latency.

Evaluates vision models on prosthesis-specific video understanding, including instance segmentation of amputees and prosthetic limbs, 2D human pose estimation with focus on lower-body keypoints, and automated gait pattern classification from pose sequences. Use when the user wants to benchmark on ProGait, or asks about evaluating this task. Reports mIoU, AP@[0.5,0.95].

This protocol evaluates the causal impact of specific profile image features (smile, body-shot, and gender) on lender selection preferences in a simulated micro-lending marketplace. It uses a conjoint-style choice experiment with GAN-generated images to isolate how visual cues influence funding decisions independent of borrower creditworthiness. Use when the user wants to benchmark on Custom GAN-generated profile images, or asks about evaluating this task. Reports Average Treatment Effect (ATE).

Evaluates reinforcement learning agents on their ability to learn efficiently and generalize to unseen, procedurally generated environments. It measures how well algorithms adapt to novel level distributions under strict computational and timestep constraints. Use when the user wants to benchmark on Procgen Benchmark, or asks about evaluating this task. Reports mean normalized return.

This evaluation probes the training efficiency and multi-GPU scaling behavior of deep learning frameworks. It measures how quickly models process mini-batches and how effectively data parallelization affects model convergence across various network architectures and hardware configurations. Use when the user has predictions and gold and needs to compute processing_time.

Evaluates multimodal foundation models on open-ended, expert-level queries across 10 professional domains, probing visual perception, domain knowledge, and long-context reasoning in single-round, multi-lingual, and multi-turn settings. Use when the user wants to benchmark on ProBench, or asks about evaluating this task. Reports ELO rating.

Evaluates real-time probabilistic forecasting of financial and weather time series, probing a model's ability to quantify uncertainty via quantile modeling and maintain calibration over sequential submission rounds. Use when the user wants to benchmark on DAX, Wind, Temperature, or asks about evaluating this task. Reports skill score.

Evaluates the skill of a probabilistic Random Forest model in forecasting severe thunderstorms (tornadoes, large hail, damaging winds) 4–8 days in advance using ensemble meteorological data. It probes the model's calibration, discrimination, and spatial coverage compared to human-generated SPC outlooks. Use when the user wants to benchmark on SPC Severe Weather Reports & GEFSv12 Reforecast, or asks about evaluating this task. Reports Brier Skill Score (BSS).

Assesses an LLM agent's ability to understand and follow privacy norms while performing real-world tasks. It measures both helpfulness and the rate at which sensitive information is incorrectly exposed. Use when the user wants to benchmark on PrivacyLens, or asks about evaluating this task. Reports privacy leakage rate.

Evaluates the trade-offs between privacy preservation, model utility, and computational/energy costs in hybrid privacy-preserving vision systems. It probes how combining federated learning with differential privacy or secure multi-party computation affects convergence, classification accuracy, and resource consumption across different neural architectures. Use when the user wants to benchmark on Alzheimer MRI Classification, ISIC Skin Lesion Classification, or asks about evaluating this task....

Evaluates large multimodal models' ability to detect, correct, and reason over real-world multimodal inconsistencies in scientific papers. It probes inter-modal mismatch detection, structured reasoning, and robustness to linguistic shortcuts versus genuine visual grounding. Use when the user wants to benchmark on PRISMM-Bench, or asks about evaluating this task. Reports Accuracy (%).

Probes LLM hallucinations across four dimensions (knowledge missing, knowledge errors, reasoning errors, and instruction-following errors) by isolating error sources through controlled, task-specific queries. It evaluates model reliability and guides optimization by measuring error rates across memory, instruction, and reasoning generation stages. Use when the user wants to benchmark on PRISM, or asks about evaluating this task. Reports H-Score.

Evaluates fine-grained, multi-aspect-aware paper-to-paper retrieval by decomposing long-form query papers into aspect-specific views and segmenting candidate papers into section-level representations for targeted retrieval. Use when the user wants to benchmark on SciFullBench, PatentFullBench, or asks about evaluating this task. Reports Recall@K.

Evaluates the ability of sentiment lexicons or models to assign accurate real-valued polarity scores to individual terms, measuring rank correlation with gold standards. Use when the user wants to benchmark on Term test set, or asks about evaluating this task. Reports Kendall's τ coefficient.

Evaluates the effectiveness of various prior-based loss functions (low-level boundary/distance and high-level shape/size constraints) for medical image segmentation across diverse anatomical structures and imaging modalities. Use when the user wants to benchmark on WMH, ISLES, Atrium, Colon, Spleen, Hippocampus, Prostate, ACDC, or asks about evaluating this task. Reports Dice score.

Evaluates large language models' ability to reason about medical ethics using the Principlism framework (autonomy, non-maleficence, beneficence, justice). It probes both theoretical knowledge of ethical principles and their practical application to complex, open-ended clinical dilemmas. Use when the user wants to benchmark on PrinciplismQA, or asks about evaluating this task. Reports Knowledge accuracy, Practice score.

Probes an LLM's ability to align generated responses with a set of natural language constitutional principles without parameter fine-tuning. It measures both overall conformance quality and the reduction of critical principle violations through an inference-time self-correction pipeline. Use when the user wants to benchmark on SafeRLHF, HH-RLHF, or asks about evaluating this task. Reports 5-Point Likert Score Ranking.