Claude Skills by qhjqhj00

Probes a model's ability to ground referring expressions in images by understanding spatial and relational context between objects. It evaluates weakly supervised region proposal scoring and context pooling mechanisms without requiring explicit bounding box annotations for context regions. Use when the user wants to benchmark on Google RefExp, UNC RefExp, or asks about evaluating this task. Reports Precision@1.

This evaluation probes a model's ability to understand visual context by localizing objects described by natural language expressions (comprehension) and generating unambiguous, context-aware descriptions for objects in an image (generation). It specifically tests whether the model leverages intra-category visual comparisons and joint language modeling to produce discriminative referring expressions. Use when the user wants to benchmark on RefCOCO, RefCOCO+, RefCOCOg, or asks about evaluating...

This benchmark evaluates a model's ability to generate referring expressions that enable humans to quickly and accurately identify a target object in an image. It prioritizes human comprehension speed and accuracy over purely semantic correctness, particularly for low-salience targets. Use when the user wants to benchmark on RefCOCO, RefCOCO+, RefCOCOg, RefGTA, or asks about evaluating this task. Reports R1-CIDEr.

Evaluates visual grounding capabilities by measuring how well a model segments or localizes objects in images based on natural language descriptions. It probes the model's ability to handle ambiguous references, diverse textual forms, and generalized referring expressions without task-specific decoders. Use when the user wants to benchmark on RefCOCO, RefCOCO+, RefCOCOg, gRefCOCO, or asks about evaluating this task. Reports mIoU.

Evaluates a model's ability to perform dense grounded understanding by localizing and segmenting specific objects in images and videos based on natural language instructions or referring expressions. Use when the user wants to benchmark on Ref-SAV, RefCOCO, RefCOCO+, RefCOCOg, MeVIS, Ref-YTVOS, ReVOS, or asks about evaluating this task. Reports cIoU.

Evaluates a model's ability to generate unambiguous, context-aware text descriptions for specific objects in an image, and to comprehend those descriptions by correctly localizing the target object via bounding box prediction. Use when the user wants to benchmark on G-Ref, UNC-Ref, or asks about evaluating this task. Reports precision@1.

This benchmark evaluates a model's ability to perform relation extraction on complex, domain-specific financial documents (SEC 10-X filings). It specifically probes challenges such as numerical inference, semantic ambiguity between similar relation types, and directional dependency resolution in long-range financial text. Use when the user wants to benchmark on REFiND, or asks about evaluating this task. Reports micro-F1.

Evaluates the zero-shot generalization capability of autoregressive language models across multiple task aggregates. It measures how well models trained on raw web data perform on downstream tasks without any fine-tuning or prompt engineering. Use when the user wants to benchmark on Eleuther AI LM evaluation harness (zero-shot aggregates), or asks about evaluating this task. Reports zero-shot accuracy.

Evaluates single-view 3D stereo reconstruction quality in scenes containing mirror reflections. It probes a model's ability to leverage virtual views generated from mirror reflections to recover accurate 3D geometry and camera poses, outperforming standard monocular or stereo baselines that typically hallucinate depth or collapse in reflective regions. Use when the user wants to benchmark on Synthetic Dataset (Reflect3r), Real-world Mirror Scenes, or asks about evaluating this task. Reports F...

Evaluates NLP models on summarizing student course reflections across three tasks: document selection, phrase extraction with support counts, and abstractive summarization. It probes specificity-aware summarization capabilities and model robustness in low-resource educational settings with variable text structure. Use when the user wants to benchmark on ReflectSumm, or asks about evaluating this task. Reports Unspecified in text.

Probes the ability of an automated evaluation framework to assess the quality of machine-generated log summaries without relying on human-written reference texts. It measures how well LLM-based judgment correlates with human preferences across dimensions like relevance, informativeness, and coherence. Use when the user has predictions and gold and needs to compute REFLEX.

Evaluates a model's ability to segment specific objects in remote sensing images guided by natural language expressions, with a focus on accurately localizing small, scattered targets that are characteristic of aerial and satellite imagery. Use when the user wants to benchmark on RefSegRS, or asks about evaluating this task. Reports mIoU.

Evaluates NLP models' ability to perform legal information extraction (NER) and predict refugee claim decision outcomes from Canadian legal documents. It probes the model's capacity to handle domain-specific terminology, extract structured entities from unstructured text, and classify case outcomes based on judicial reasoning. Use when the user wants to benchmark on Canadian Refugee Status Determination (RSD) Cases, or asks about evaluating this task. Reports accuracy.

Evaluates conversational recommender systems on next-item prediction and joint narrative generation, specifically testing how well models incorporate user interaction history and explicit natural language critiques to produce accurate recommendations and contextually grounded textual explanations. Use when the user wants to benchmark on REGEN, or asks about evaluating this task. Reports Recall@10.

Evaluates whether increasing the length of a user's purchase history context improves recommendation quality for LLM-based agents. It probes the saturation point of personalization reasoning and the cost-efficiency trade-off of context length. Use when the user wants to benchmark on REGEN, or asks about evaluating this task. Reports quality scores.

Evaluates the ability of vision-language models to generate accurate and semantically rich chest X-ray radiology reports from medical images. It tests both lexical overlap and clinical semantic alignment of the generated findings and impressions against ground-truth reports. Use when the user wants to benchmark on ReXGradient-160K, or asks about evaluating this task. Reports COMET.

Evaluates online resource allocation algorithms by measuring their cumulative regret and cumulative unfairness across simulated environments with varying resource binding and degeneracy conditions. Use when the user has predictions and gold and needs to compute cumulative_unfairness.

This benchmark evaluates deep learning models on skeleton-based human motion rehabilitation assessment. It probes both classification (e.g., motion type or health status) and extrinsic regression tasks using standardized cross-subject splits and min-max normalization to prevent data leakage. Use when the user wants to benchmark on Rehab-Pile, or asks about evaluating this task. Reports accuracy.

This benchmark probes an LLM's ability to perform high-arity relational reasoning and multi-constraint integration across scientific domains. It isolates the difficulty of jointly binding independent entities to satisfy a relation, independent of prompt length or in-context learning. Use when the user wants to benchmark on REL, or asks about evaluating this task. Reports accuracy.

Evaluates the effectiveness of machine-learned preconditioners for linear solvers in semi-implicit shallow-water models by measuring error reduction in the first solver iteration and overall convergence rates during free-running simulations. Use when the user has predictions and gold and needs to compute Relative decrease in MAE.

Evaluates the prediction accuracy of data-driven neural network closures for kinetic theory of active fluids against reference kinetic simulations and traditional analytical closures. It probes the model's ability to capture rotational symmetries, generalize across parameter regimes, and resolve topological defects in fluid dynamics. Use when the user has predictions and gold and needs to compute RMSE.

Evaluates the precision of estimating globular cluster distance and mass parameters by quantifying how much gravitational wave modulation reduces parameter uncertainties compared to electromagnetic-only observations. Use when the user has predictions and gold and needs to compute relative_uncertainty.

Compute the RelativeAverageSpectralError metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute RelativeAverageSpectralError, or asks how to score with RelativeAverageSpectralError.

Compute the RelativeSquaredError metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute RelativeSquaredError, or asks how to score with RelativeSquaredError.

Evaluates a model-agnostic fairness calibration heuristic that adjusts prediction thresholds per protected attribute value to independently control false positive and false negative rates. It probes the ability to balance business-critical error costs across high-arity and multiple sensitive groups while maintaining predictive performance. Use when the user wants to benchmark on COMPAS (Criminal Recidivism), Income-Prediction, Health Prediction, Proprietary Online Fraud Dataset, or asks about...

This protocol evaluates the reliability, consistency, and inter-correlation of various open-ended and close-ended evaluation metrics on healthcare LLM outputs. It specifically probes how well metrics capture factual coherence and content quality while being robust to output rephrasing and sampling variations. Use when the user has predictions and gold and needs to compute Relaxed Perplexity.

Evaluates the ability of graph neural networks to learn from relational databases by predicting entity attributes (classification and regression) and discovering relationships between entities (link prediction) using primary-foreign key graph structures. Use when the user wants to benchmark on RELBENCH, or asks about evaluating this task. Reports AUROC.

Evaluates whether pointwise re-rankers can function as binary relevance judges by predicting whether a document is relevant to a query. It probes the capability of adapted ranking models to perform direct relevance classification and compares their performance against LLM-based judges. Use when the user wants to benchmark on TREC-DL, or asks about evaluating this task. Reports binary accuracy.

Compute Remeris/rouge_ru via the HuggingFace `evaluate` library. Use when the user has predictions + references and wants the canonical implementation of Remeris/rouge_ru.

Evaluates vision-language models' ability to perform multi-view spatial reasoning, including relative direction, relative distance, cross-view consistency, and perspective-taking. It probes whether models can maintain geometric coherence and integrate information across multiple camera viewpoints to answer questions about procedurally generated indoor scenes. Use when the user wants to benchmark on ReMindView-Bench, or asks about evaluating this task. Reports accuracy.

Evaluates in-domain representation learning and scene classification across diverse remote sensing modalities (optical, SAR, aerial) and spatial resolutions. It probes model robustness to varying class balances, visual similarities, and label domains in Earth observation. Use when the user wants to benchmark on BigEarthNet, EuroSAT, RESISC-45, So2Sat, UC Merced, or asks about evaluating this task. Reports accuracy.

Probes the robustness and cross-condition consistency of multimodal large language models on Earth observation tasks under realistic visual and textual perturbations. Evaluates performance degradation and behavioral stability across clean and perturbed inputs for scene classification, VQA, and visual grounding. Use when the user wants to benchmark on RemoteShield clean-perturbed benchmarks, or asks about evaluating this task. Reports RPD, CCA.

Evaluates session-based recommendation models on predicting the next item in a user session. It probes the model's ability to capture sequential patterns and handle repeat consumption behaviors across e-commerce and music domains. Use when the user wants to benchmark on YOOCHOOSE, DIGINETICA, LASTFM, or asks about evaluating this task. Reports Recall@k.

Evaluates the robustness of audio deepfake detection models against replay attacks where deepfake audio is played back and re-recorded through real-world hardware, introducing acoustic distortions and room impulse responses. Use when the user wants to benchmark on ReplayDF, or asks about evaluating this task. Reports EER (%).

Evaluates LLMs' ability to read unseen reference documents and answer questions based solely on the provided context, as well as their ability to detect unanswerable questions and classify document topics. It specifically probes whether models rely on pre-training memory versus actual context-conditional reading and retrieval skills. Use when the user wants to benchmark on RepLiQA, TriviaQA, or asks about evaluating this task. Reports recall.

Evaluates autoregressive language models on predicting the next line of code using provided in-file and cross-file contexts. Use when the user wants to benchmark on RepoBench-C, or asks about evaluating this task. Reports Exact Match (EM).

Evaluates an end-to-end pipeline that first retrieves cross-file snippets and then predicts the next line of code using both the in-file context and retrieved snippets. Use when the user wants to benchmark on RepoBench-P, or asks about evaluating this task. Reports Exact Match (EM).

Evaluates a model's ability to retrieve relevant cross-file code snippets given an in-file context for predicting the next line of code. Use when the user wants to benchmark on RepoBench-R, or asks about evaluating this task. Reports acc@1.

This benchmark evaluates repository-level code completion by measuring how accurately a model predicts missing code segments given surrounding context and retrieved repository snippets. It probes both syntactic similarity and functional correctness across line, API, and function-level granularity. Use when the user wants to benchmark on RepoEval, or asks about evaluating this task. Reports Exact Match (EM).

Evaluates how input representation choices—quantization granularity, value encoding, temporal encoding, and vocabulary remapping—affect downstream predictive performance on clinical outcomes. It probes the model's ability to extract and utilize structured medical event sequences for binary classification and regression tasks. Use when the user wants to benchmark on MIMIC-IV, or asks about evaluating this task. Reports AUROC.

Evaluates whether agentic AI systems can accurately assess the computational reproducibility of social science research by comparing original paper findings against results reproduced from provided raw data and code. It probes end-to-end agentic reasoning, including command execution, debugging, and result interpretation in a simulated research environment. Use when the user wants to benchmark on REPRO-Bench, or asks about evaluating this task. Reports accuracy.

Evaluates the ability of LLMs and AI agents to automatically repair broken R-based social science code and restore computational reproducibility. It probes how well different workflows handle varying error complexities and contextual information. Use when the user wants to benchmark on Custom R-based Social Science Code Dataset, or asks about evaluating this task. Reports reproduction_success_rate.

Evaluates the ability of embedding and reranking models to retrieve and rank relevant paragraphs from descriptive linguistic grammars based on typological feature queries, specifically testing their capacity to filter noisy or partially relevant context. Use when the user wants to benchmark on The Benchmark for Rerankers, or asks about evaluating this task. Reports NDCG@k.

Evaluates the capability of LLM agents to conduct closed-loop scientific research by proposing hypotheses, executing experiments, and outperforming human baselines on repurposed real-world AI papers. It probes long-horizon planning, resource management, and autonomous experimentation under realistic tool constraints. Use when the user wants to benchmark on ResearchGym, or asks about evaluating this task. Reports improvement over baselines.

Evaluates whether a multi-agent research simulator can accurately reconstruct masked research nodes (papers and reviews) from their local neighborhood context in a collaborative graph. It probes the model's ability to capture interdisciplinary collaboration patterns and realistic academic writing styles. Use when the user wants to benchmark on ResearchTown simulated community graph, or asks about evaluating this task. Reports reconstruction_similarity.

Evaluates a model's ability to perform N-ary relation extraction from scientific documents by first retrieving relevant text/table components and then selecting the correct entities within those components. Use when the user wants to benchmark on SciREX, PubMed, NLP-TDMS (Full), or asks about evaluating this task. Reports Accuracy (Acc).

Evaluates a machine learning model's ability to predict the probability of hydrocarbon reservoir presence in a 3D geological space using seismic and well log data. It probes the model's capacity for binary lithological classification and probabilistic calibration under early-stage exploration conditions with limited well data. Use when the user wants to benchmark on Achimov sedimentary complex field dataset, or asks about evaluating this task. Reports classification quality.

Evaluates an integer linear programming model for allocating network monitoring depth across protocol layers, balancing detection efficiency against computational resource constraints on a synthetic heterogeneous network. Use when the user wants to benchmark on Synthetic 6-Device Network, or asks about evaluating this task. Reports objective function value.

This evaluation protocol measures the computational efficiency and energy consumption of distributed deep learning training runs. It probes how model architecture, dataset, and hardware constraints (GPU count, power caps, clock speeds) affect training speed and resource utilization. Use when the user wants to benchmark on ImageNet, WikiText-103, QM9, or asks about evaluating this task. Reports training speed.

This benchmark evaluates large language models on bilingual Latin-English question answering across knowledge-based, skill-based (grammar, scansion, literary devices), multihop reasoning, and translation tasks. It probes models' ability to handle classical language morphology, poetic meter analysis, and cross-lingual generation under constrained and unconstrained settings. Use when the user wants to benchmark on RespondeoQA, or asks about evaluating this task. Reports exact-match accuracy.