Claude Skills by qhjqhj00

Probes a model's ability to perform compositional visual reasoning and ground referring expressions in the presence of semantically similar distractors. It evaluates whether models can parse complex logical structures and distinguish fine-grained visual differences rather than relying on statistical biases. Use when the user wants to benchmark on Cops-Ref, or asks about evaluating this task. Reports accuracy.

Evaluates the robustness of copyright tracking methods in fine-tuned Large Vision-Language Models (LVLMs) by measuring whether adversarial image triggers can consistently elicit a predefined target response after the model has been adapted on various downstream datasets. Use when the user wants to benchmark on ImageNet 2012 (validation subset), V7W, ST-VQA, TextVQA, PaintingForm, MathV360k, ChEBI-20, or asks about evaluating this task. Reports target match rate (TMR).

Evaluates a model's ability to answer free-form questions in a multi-turn conversational setting. It probes coreference resolution, pragmatic reasoning, and the capacity to maintain and leverage dialogue history over a given context passage. Use when the user wants to benchmark on CoQA, or asks about evaluating this task. Reports macro-average F1 score of word overlap.

Evaluates a language model's ability to synthesize complete formal proofs in Coq by dynamically retrieving relevant project-specific lemmas and proofs. It measures how effectively retrieval-augmented proving and search strategies improve theorem synthesis success rates over time. Use when the user wants to benchmark on CoqStoq, or asks about evaluating this task. Reports Theorems Proven.

Evaluates continual reinforcement learning agents across sequential task environments, probing their ability to learn new tasks while retaining old ones (plasticity vs stability) and generalizing to unseen contexts. Use when the user wants to benchmark on Procgen, MiniHack, CHORES, Atari, or asks about evaluating this task. Reports Continual Evaluation ($\mathcal{C}$).

Probes semantic segmentation models on complex underwater scenes characterized by high morphological variability, degradation states, and visual distortions. It evaluates the model's ability to generalize across geographically distinct reef sites and handle severe class imbalance and fine-grained benthic classification. Use when the user wants to benchmark on Coralscapes, or asks about evaluating this task. Reports mean Intersection over Union (mIoU).

Evaluates a collaborative reranking framework that combines a small efficient reranker with a large LLM-based reranker. It uses a reinforcement learning-trained passage order adjuster to mitigate positional bias and reduce latency while maintaining ranking effectiveness on standard IR benchmarks. Use when the user wants to benchmark on TREC DL (DL19, DL20), BEIR (TREC-Covid, Robust04, Trec-News), BRIGHT (Economics, Earth Science, Robotics), or asks about evaluating this task. Reports NDCG@10.

Evaluates parameter-efficient fine-tuning methods across mathematical reasoning, code generation, instruction following, and general language understanding tasks, while measuring their ability to retain pre-trained world knowledge. Use when the user wants to benchmark on MetaMathQA, GSM8k, Math, CodeFeedback, HumanEval, MBPP, WizardLM-Evol-Instruct, MTBench, TriviaQA, NQ open, WebQS, GLUE, Wikitext-2, Penn TreeBank (PTB), or asks about evaluating this task. Reports exact match scores.

Evaluates foundational language, reasoning, instruction-following, and multimodal understanding capabilities across text, images, charts, documents, and video, alongside multilingual translation and agentic function-calling proficiency. Use when the user wants to benchmark on MMLU, GSM8K, MATH, IFEval, Flores200, ChartQA, DocVQA, TextVQA, Berkeley Function Calling Leaderboard (BFCL), or asks about evaluating this task. Reports exact match accuracy.

Evaluates few-shot relation classification models on company and business entity relations, testing their ability to resolve entity ambiguity and adapt across domains using limited labeled examples. Use when the user wants to benchmark on CORE, or asks about evaluating this task. Reports Micro F1.

Evaluates multimodal fusion models for robust brain tumor typing by integrating MRI, histopathology, and diagnostic text under variable modality availability conditions. The benchmark probes a model's ability to perform fine-grained hierarchical classification across six glioma subtypes when some modalities are missing or degraded. Use when the user wants to benchmark on CoRe-BT, or asks about evaluating this task. Reports accuracy.

Evaluates a model's ability to detect corporate fraud using financial graphs. It specifically probes robustness to information overload from noisy support nodes (e.g., directors) and label noise caused by delayed fraud detection. Use when the user wants to benchmark on MBM, SME, GEM, or asks about evaluating this task. Reports AUC.

Evaluates the structural integrity, metadata completeness, and text quality of a legally screened chemistry corpus derived from S2ORC. It verifies schema compliance, metadata field availability, subfield label validity, chunking consistency, and embedding reproducibility against predefined thresholds. Use when the user wants to benchmark on Lit2Vec Chemistry Corpus, or asks about evaluating this task. Reports schema_pass_rate.

Evaluates whether turn-level conversational metrics in LLM interactions suffer from temporal autocorrelation that inflates statistical significance. It compares naive pooled hypothesis testing against cluster-robust corrections to measure false positive rates and classify metric robustness. Use when the user has predictions and gold and needs to compute corrected_p_value.

Evaluates how closely an LLM's legal interpretations align with different parties (Applicant, Court, State) in Italian constitutional bioethics cases, and measures the consistency of this alignment across multiple prompt iterations. It probes value alignment, legal reasoning capability, and robustness to prompt variations in complex, value-sensitive scenarios. Use when the user wants to benchmark on Italian Constitutional Bioethics Case Dataset, or asks about evaluating this task. Reports cos...

Compute the CosineSimilarity metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute CosineSimilarity, or asks how to score with CosineSimilarity.

Evaluates the ability of graph neural networks to extract cosmological parameters and local velocity fields from large-scale 3D point clouds of dark matter halos. It probes both global long-range correlation capture (via cosmological parameter regression) and local geometric dependency modeling (via per-node velocity prediction). Use when the user wants to benchmark on Quijote BSQ Point Cloud, or asks about evaluating this task. Reports MSE.

Evaluates the compute efficiency and horizontal scalability of a 3D convolutional neural network framework on supercomputers, measuring sustained floating-point throughput and parallel scaling efficiency across thousands of nodes. Use when the user has predictions and gold and needs to compute Pflop/s.

Evaluates the effectiveness of a synthetic driving data generation pipeline by measuring performance gains in downstream autonomous driving perception tasks, including 3D lane detection, 3D object detection, and LiDAR-based detection, particularly under challenging conditions like extreme weather and nighttime. Use when the user wants to benchmark on Waymo Open Dataset, RDS-HQ, RDS-HQ-HL, or asks about evaluating this task. Reports F1-score.

This benchmark evaluates a model's ability to perform contextual commonsense reasoning in machine reading comprehension. It probes whether systems can make non-literal, implicit inferences about causes, effects, and counterfactuals based on personal narratives, rather than relying on explicit textual evidence or simple semantic matching. Use when the user wants to benchmark on Cosmos QA, or asks about evaluating this task. Reports Accuracy.

This benchmark evaluates a model's ability to detect malware within Docker container file systems by treating the entire file system as a large-scale RGB image. It probes the capability to identify subtle, localized malicious byte patterns against a largely benign background, simulating real-world container security threats. Use when the user wants to benchmark on COSOCO, or asks about evaluating this task. Reports F1 score.

This benchmark probes a model's ability to perform compositional generalization in context-dependent Text-to-SQL. It evaluates whether models can correctly combine previously seen SQL query structures with novel modification patterns (e.g., new WHERE or ORDER BY clauses) in multi-turn dialogues. Use when the user wants to benchmark on CoSQL-CG, or asks about evaluating this task. Reports question match (QM).

Evaluates conversational text-to-SQL systems on cross-domain database querying. It probes dialogue state tracking via SQL grounding, response generation from query results, and user intent/dialogue act prediction under real-world ambiguity and clarification dynamics. Use when the user wants to benchmark on CoSQL, or asks about evaluating this task. Reports Question Match.

This evaluation probes a containerized framework's capability to automate the deployment and execution of distributed big data workloads. It specifically measures how execution time and monetary cloud costs scale as the cluster size increases from four to eight nodes. Use when the user has predictions and gold and needs to compute cost.

Economic viability and cost-aware performance of embodied agents in urban sidewalk delivery navigation. It evaluates how technical metrics like collision rate and arrival success translate into real-world financial outcomes, including maintenance costs, energy usage, revenue, and break-even points. Use when the user wants to benchmark on CostNav Urban Sidewalk Navigation Simulation, or asks about evaluating this task. Reports Profit/run.

Evaluates zero-shot text-to-speech synthesis quality, focusing on content consistency (how well generated speech matches input text) and speaker similarity (how well the cloned voice matches the reference speaker) across English and Chinese. It also probes emotion controllability and the utility of synthesized speech for augmenting ASR training data. Use when the user wants to benchmark on LibriTTS, AISHELL-3, or asks about evaluating this task. Reports WER (%), CER (%).

Evaluates the zero-shot and few-shot reasoning capabilities of language models, specifically probing their ability to generate step-by-step chain-of-thought rationales and produce correct answers across classification and generation tasks. Use when the user wants to benchmark on BigBench Hard (BBH), P3, MGSM, or asks about evaluating this task. Reports accuracy.

Evaluates whether reasoning models explicitly acknowledge external hint injections within their chain-of-thought reasoning traces. It probes model transparency and the alignment between internal reasoning tokens and final output disclosures. Use when the user wants to benchmark on MMLU, GPQA Diamond, or asks about evaluating this task. Reports faithfulness.

Evaluates how well AI models can assist human moderators in content moderation by prioritizing comments for review. It probes the model's ability to estimate uncertainty accurately and guide human review capacity to maximize collaborative accuracy and efficiency under constraints. Use when the user wants to benchmark on CoToMoD, or asks about evaluating this task. Reports OC-Acc.

Evaluates a model's ability to predict the next item in a user session based on historical click sequences. It probes the model's capacity to capture sequential dependencies and session-level patterns in sparse e-commerce or media interaction data. Use when the user wants to benchmark on Tmall, RetailRocket, Diginetica, or asks about evaluating this task. Reports P@10.

Evaluates a multi-agent consensus framework's ability to mitigate hallucinations and biases in large language models compared to individual frontier models. It probes factual accuracy, truthfulness, informativeness, and consistency across diverse knowledge domains and varying reasoning complexities. Use when the user wants to benchmark on HaluEval, TruthfulQA, Multi-Domain Reasoning, or asks about evaluating this task. Reports Hallucination Rate.

Evaluates the human-centric quality of counterfactual explanations across eight explanatory virtues. It probes how well explanations convey desired outcomes, remain feasible, consistent, complete, trustworthy, understandable, fair, and appropriately complex. Use when the user wants to benchmark on CounterEval, or asks about evaluating this task. Reports Overall Satisfaction, Feasibility, Consistency, Completeness, Trust, Understandability, Fairness, Complexity.

Evaluates the effectiveness and stability of sequential LLM knowledge editing methods. It measures how well a model updates a specific fact while preserving related paraphrases, neighboring facts, generation fluency, and overall general capabilities over thousands of edits. Use when the user wants to benchmark on CounterFact, GLUE_MMLU_GSM8K_HumanEval_MBPP, or asks about evaluating this task. Reports Efficacy.

Evaluates the reliability of counterfactual trajectory estimation in chaotic versus non-chaotic dynamical systems under parameter uncertainty and observational noise. It probes whether Bayesian filtering and particle-based smoothing can accurately recover 'what-if' scenarios when small initial perturbations lead to divergent outcomes. Use when the user wants to benchmark on Lorenz System, Rössler System, Logistic Growth, or asks about evaluating this task. Reports RMSE_t.

Evaluates a model's ability to detect counterfactual statements in product reviews. It probes robustness to selection bias from clue phrases, cross-lingual transfer via machine translation, and the effectiveness of different sentence encoders and classifiers on imbalanced binary classification tasks. Use when the user wants to benchmark on Multilingual Counterfactual Detection Dataset (Amazon Reviews), or asks about evaluating this task. Reports F1.

Evaluates the trade-off between predictive accuracy and counterfactual fairness on real-world datasets. It measures how well a model's predictions remain invariant to sensitive attributes (race, gender) while maintaining performance on regression or classification tasks. Use when the user wants to benchmark on LSAC, Compas, Adult, or asks about evaluating this task. Reports Balanced Accuracy.

Evaluates whether LLM-based contact center QA systems exhibit systematic bias when agent identity (gender, ethnicity, religion, disability) or contextual factors (past performance, behavioral style) are counterfactually altered. Measures if model judgments change disproportionately based on these attributes rather than transcript content. Use when the user wants to benchmark on Contact-Center QA Transcripts, or asks about evaluating this task. Reports Counterfactual Flip Rate (CFR).

Evaluates the capability of generative models to produce counterfactual images that preserve causal consistency, maintain realism, and minimally alter non-target attributes under specified interventions. It probes composition stability, attribute manipulation effectiveness, and distributional fidelity across varying dataset complexities and causal graphs. Use when the user wants to benchmark on MorphoMNIST, CelebA, ADNI, or asks about evaluating this task. Reports FID, CLD.

Evaluates how well counterfactual representations (CFRs) in high-dimensional embedding space mimic true text counterfactuals, measuring prediction consistency, probability alignment, and downstream fairness improvements across synthetic and real-world biased datasets. Use when the user wants to benchmark on EEEC+, BiasInBios, or asks about evaluating this task. Reports PIP.

This metric quantifies the promotional or visionary framing of a machine learning paper's abstract, isolating rhetorical style from the underlying technical content. It uses counterfactual abstracts generated by diverse LLM personas and pairwise comparisons to produce a calibrated continuous score of rhetorical strength. Use when the user has predictions and gold and needs to compute Rhetorical Score.

Evaluates a fairness auditing framework's ability to detect individual discrimination in decision-making systems by comparing factual outcomes against counterfactual or similar-group outcomes. It probes whether protected attributes causally influence decisions beyond legitimate factors. Use when the user wants to benchmark on Synthetic Loan Application, Law School Admissions, or asks about evaluating this task. Reports individual discrimination cases.

This benchmark evaluates the effectiveness and linguistic quality of counterfactual text generation methods. It probes a model's ability to modify input text to flip a target classifier's predicted label while preserving grammatical correctness, fluency, and coherence, highlighting the trade-off between label-flipping success and text quality. Use when the user wants to benchmark on IMDB, SNLI, or asks about evaluating this task. Reports flip rate (FR).

This benchmark evaluates the object counting and spatial individuation capabilities of multimodal large language models (MLLMs) on real-world images characterized by high density, clutter, and occlusion. It probes whether generalist models can perform precise, fine-grained visual grounding and numerical reasoning out-of-the-box without specialized training. Use when the user wants to benchmark on CountQA, or asks about evaluating this task. Reports Exact Match (EM).

Tests a model's ability to predict binary case outcomes (accepted/denied) and generate human-readable explanations by citing relevant sentences from the input document. This probes joint reasoning, outcome forecasting, and justification generation in legal contexts. Use when the user wants to benchmark on LegalEval CJPE Dataset, or asks about evaluating this task. Reports standard F1 score.

Evaluates LLM safety guardrails and policy-adaptation frameworks on their ability to correctly identify harmful, toxic, or policy-violating content across diverse attack vectors. It probes robustness against automated jailbreaks, over-refusal in benign contexts, and zero-shot adaptability to out-of-domain policy enforcement. Use when the user wants to benchmark on AdvBenchM, WildGuard, HarmBench, JailJudge, PKU-SafeRLHF, ToxicChat, BeaverTails, XSTest, PAN Wikipedia Vandalism Corpus 2010, Hum...

Compute the coverage_error metric — provided by scikit-learn. Use when the user has predictions and ground-truth and needs to compute coverage_error, or asks how to score with coverage_error.

Measures the alignment between a proxy benchmark's ranking and a target reward modeling benchmark's ranking at the top-k positions. It quantifies how many of the highest-performing models on a reward benchmark are also identified as top performers on a given proxy benchmark. Use when the user has predictions and gold and needs to compute coverage_at_top_k.

This benchmark evaluates AI models for detecting COVID-19 infection and assessing lung severity using lung ultrasound videos, clinical variables, and blood count data. It measures how well zero-shot and fine-tuned models generalize to real-world, heterogeneous clinical data compared to human annotators and tabular baselines. Use when the user wants to benchmark on COVID-BLUeS, or asks about evaluating this task. Reports accuracy.

Evaluates the impact of five image enhancement techniques (histogram equalization, CLAHE, complement, gamma correction, BCET) on six CNN architectures for three-class classification (COVID-19, lung opacity, normal) using chest X-ray images. It also assesses whether lung segmentation improves classification accuracy and model interpretability. Use when the user wants to benchmark on COVQU-20, or asks about evaluating this task. Reports Accuracy.

This benchmark evaluates deep graph generative models (JT-VAE and DQN) for their ability to design novel molecular structures optimized for high predicted potency against the SARS-CoV-2 3CL-protease, while balancing drug-likeness, lipophilicity, and synthesizability. It also assesses structural novelty relative to known antivirals and predicted binding affinity using in silico classifiers. Use when the user wants to benchmark on ChEMBL/BindingDB/ToxCat pharmacology dataset, or asks about eval...