Claude Skills by qhjqhj00

Evaluates the effectiveness of a training-free token compression method for video large language models across various video understanding tasks, including QA, long-video understanding, and multi-choice benchmarks, under different token retention ratios. Use when the user wants to benchmark on ActivityNet-QA, Video-ChatGPT, Next-QA, Egoschema, MLVU, Video-MME, VideoMMMU, MVBench, or asks about evaluating this task. Reports Avg.(%).

Evaluates the ability of LLMs to accurately classify academic papers as discussing LLM limitations and to extract supporting evidence from abstracts. It measures alignment with human expert annotations using ordinal rating agreement and span-level extraction metrics. Use when the user wants to benchmark on ACL Anthology & arXiv (crawled 2022-2025), or asks about evaluating this task. Reports weighted-cohens-kappa.

Evaluates the ability of large language models to automatically generate correct code patches for buggy functions across Java, JavaScript, Python, and PHP. It probes language-specific repair capabilities, the impact of providing test case information, and the sensitivity to fault localization granularity. Use when the user wants to benchmark on Defects4J, BugsInPy, BugsJS, BugsPHP, or asks about evaluating this task. Reports plausible@1.

Evaluates a language model's general capabilities, including in-context learning, instruction following, mathematical reasoning, code generation, and bidirectional reversal reasoning across a suite of standard and custom benchmarks. Use when the user wants to benchmark on MMLU, GSM8K, HumanEval, Chinese Poem Sentence Pairs, or asks about evaluating this task. Reports accuracy.

This evaluation probes the propensity of large language models to generate stereotypical or biased predictions across multiple demographic and social categories. It measures how often models align with human-annotated stereotypes versus anti-stereotypes or neutral alternatives when completing masked sentences or answering repurposed benchmark questions. Use when the user wants to benchmark on StereoSet, WinoBias, UnQover, CrowS-Pairs, Real Toxicity Prompts (RTP), Equity Evaluation Corpus (EEC...

This evaluation probes whether explicitly unbiased LLMs exhibit automatic, stereotype-driven preferences in decision-making scenarios. It measures implicit bias by comparing model agreement rates across stereotypical versus counter-stereotypical social categories (e.g., gender-career, race-health) using a single-choice decision prompt rather than a relative comparison. Use when the user wants to benchmark on LLM Decision Bias (Absolute Variant), or asks about evaluating this task. Reports nor...

Evaluates how reducing the size of the language model component in multimodal models impacts task performance, specifically isolating and measuring the bottlenecks in visual perception versus logical reasoning across multiple benchmarks. Use when the user wants to benchmark on Grounding, NIGHTS, PieAPP, OCR-VQA, Fine-grained Perception, Logical Reasoning, Math, Science & Technology, or asks about evaluating this task. Reports performance.

Evaluates the utility of LLM text generation across code, math, and summarization tasks under inference budget constraints. It probes whether jointly tuning generation hyperparameters (e.g., temperature, top-p, number of responses) improves task performance compared to default or benchmark configurations. Use when the user wants to benchmark on APPS, HumanEval, MATH, XSum, or asks about evaluating this task. Reports pass_rate (code).

Evaluates LLM inference efficiency by measuring prefilling latency (TTFT), decoding latency per token (TPOT), end-to-end latency (TTLT), and corresponding energy consumption (J/Prompt, J/Token, J/Request) across varying prompt lengths, batch sizes, and hardware platforms. Use when the user has predictions and gold and needs to compute TTFT.

Evaluates Large Language Models' proficiency in Knowledge Graph Engineering tasks, specifically focusing on RDF syntax repair, SPARQL query generation and semantics, and data serialization format handling across multiple graph structures. Use when the user wants to benchmark on LLM-KG-Bench 3.0, or asks about evaluating this task. Reports capability compass.

Evaluates LLMs across 44 existing tasks to uncover latent cognitive skills using psychometric factor analysis, rather than relying on aggregated benchmark scores. It probes whether models possess coherent, interpretable skill profiles across diverse domains like reading comprehension, mathematical reasoning, and ethical judgment. Use when the user wants to benchmark on SQuAD, GSM8K, GPQA, TriviaQA, XSum, MNLI (textual entailment), Ethical/Social Judgment datasets, or asks about evaluating thi...

This evaluation protocol assesses the performance of an on-device LLM inference framework (llm.npu) on mobile NPUs. It probes the system's ability to accelerate prefill and decoding stages, manage quantization without accuracy loss, and optimize energy efficiency across various LLM sizes and real-world task datasets. Use when the user wants to benchmark on LAMBADA, HellaSwag, WinoGrande, MMLU, LongBench, DroidTask, Persona-Chat, or asks about evaluating this task. Reports end-to-end latency.

Evaluates whether an LLM-based pairwise comparison framework can effectively identify high-impact academic papers compared to human peer review and traditional rating-based LLM methods. It probes the system's predictive accuracy for future scholarly influence, decision consistency with human committees, and susceptibility to biases in topic novelty and institutional representation. Use when the user wants to benchmark on OpenReview Conference Papers (ICLR, NeurIPS, CoRL, EMNLP), or asks about...

Evaluates memory-efficient gradient compression optimizers against full-rank baselines during LLM pre-training and fine-tuning, measuring final model quality, convergence speed, memory footprint, and training throughput. Use when the user wants to benchmark on C4, MMLU, GLUE, or asks about evaluating this task. Reports Validation PPL, Accuracy.

Evaluates LLM-based query expansion methods for evidence retrieval and claim verification. It measures retrieval quality and final verdict accuracy, while also analyzing whether generated documents contain sentences entailed by ground-truth evidence to detect knowledge leakage. Use when the user wants to benchmark on FEVER, SciFact, AVeriTeC, or asks about evaluating this task. Reports Recall@5, F1.

This evaluation protocol assesses the mathematical reasoning and multi-step problem-solving capabilities of LLMs after reinforcement learning fine-tuning. It measures how well models generalize from a specialized arithmetic training task to standard academic and competitive math benchmarks. Use when the user wants to benchmark on GSM8K, BBH, MATH, MMLU-Pro, or asks about evaluating this task. Reports accuracy.

Evaluates the performance and behavioral characteristics of Large Language Models when deployed as recommender systems. It probes traditional recommendation accuracy and novelty, alongside LLM-specific traits like history length sensitivity, candidate position bias, and hallucination rates. Use when the user wants to benchmark on Unspecified recommendation datasets (four datasets referenced in paper), or asks about evaluating this task. Reports HR.

Evaluates LLM safety and robustness against adversarial attacks by measuring overall safety scores, attack success rates, and toxicity levels. It also assesses whether safety alignment preserves general capabilities across standard reasoning, instruction-following, and knowledge benchmarks. Use when the user wants to benchmark on ALERT, LLM Leaderboard, or asks about evaluating this task. Reports Safety Score S.

Evaluates a model's ability to answer open-ended protein questions accurately and predict Enzyme Commission (EC) numbers hierarchically. It probes semantic understanding of biological knowledge and fine-grained functional classification across multiple taxonomic levels. Use when the user has predictions and gold and needs to compute LLM-Score, Hierarchical Micro-F1.

Evaluates the end-to-end latency and answer accuracy of LLM-based search agents operating in a ReAct workflow with external Wikipedia API calls. It probes the system's ability to balance speculative action execution with verification to reduce inference time while maintaining multi-hop reasoning quality. Use when the user wants to benchmark on HotPotQA, 2WikiMultihopQA, TriviaQA, or asks about evaluating this task. Reports Accuracy.

Evaluates the intrinsic self-correction capability of LLMs across safety, reasoning, and vision-language tasks. It measures how iterative self-refinement reduces model uncertainty and improves calibration, toxicity mitigation, and bias reduction. Use when the user wants to benchmark on AdvBench, CommonGen-Hard, BBQ, MMVP, MS-COCO (Visual Grounding Subset), Real Toxicity Prompts, or asks about evaluating this task. Reports toxicity_score.

Evaluates the throughput, latency, and scalability of LLM inference serving systems under varying request rates and context lengths. It probes how efficiently a system manages KV cache, batching, and resource allocation for both short and long-context instruction-following workloads. Use when the user wants to benchmark on Alpaca, LongBench, or asks about evaluating this task. Reports Throughput.

Evaluates LLM harmlessness and trustworthiness across four domains: Bias, Hate, Illegal content, and Sensitiveness. It measures the model's ability to correctly identify harmful or biased prompts and respond appropriately using a multiple-choice format where safe or neutral responses are designated as correct. Use when the user wants to benchmark on LLM Trustworthiness Benchmark, or asks about evaluating this task. Reports accuracy.

This benchmark probes an evaluator model's ability to accurately judge preference between two model responses while resisting bias towards superficial qualities like verbosity, fluency, and formality. It measures instruction-following accuracy by comparing judgments on natural preference data against adversarially crafted instances designed to confound less capable judges. Use when the user wants to benchmark on LLMBar, or asks about evaluating this task. Reports accuracy.

Evaluates the effectiveness and efficiency of an LLM-based index recommendation system in selecting database indexes for given SQL workloads under varying storage constraints and schema generalization settings. Use when the user wants to benchmark on TPC-H, JOB, TPC-DS, SSAG, AMPS, or asks about evaluating this task. Reports Relative Workload Cost Reduction.

This benchmark evaluates the agreement and ranking consistency of LLM-generated relevance judgments against human assessments. It probes whether automated scoring methods can reliably replicate human relevance labels and maintain correct document ordering for information retrieval tasks. Use when the user wants to benchmark on LLMJudge test set, or asks about evaluating this task. Reports Cohen's \kappa.

Evaluates LLMs' ability to predict object entities given subject-relation pairs in Wikidata, testing knowledge retrieval, entity disambiguation, and domain-specific reasoning across 21 relations spanning 7 domains. Use when the user wants to benchmark on ISWC 2023 LM-KBC Challenge dataset, or asks about evaluating this task. Reports F1-score.

Benchmarks off-the-shelf large language models on five recommendation tasks: rating prediction, sequential recommendation, direct recommendation, explanation generation, and review summarization. It probes both accuracy-driven prediction capabilities and natural language generation for explainability, revealing gaps between objective metric scores and human-perceived quality in recommendation contexts. Use when the user wants to benchmark on LLMRec Benchmark (includes Beauty dataset), or asks...

Evaluates LLMs on ontology learning tasks including term typing, taxonomy induction, and non-taxonomic relation extraction across multiple domains and few-shot/zero-shot settings. Use when the user wants to benchmark on LLMs4OL-2024, or asks about evaluating this task. Reports F1-score.

Evaluates LLMs' capability to perform multilingual subject tagging for technical library records by ranking relevant GND taxonomy subjects based on title and abstract. It probes the model's ability to handle large-scale taxonomies, bilingual semantic processing, and customizable top-k ranking for digital library classification. Use when the user wants to benchmark on all-subjects, tib-core, or asks about evaluating this task. Reports top-k ranked list.

Evaluates large language models' ability to extract structured data from natural-language emails into valid JSON objects that adhere to a provided schema. It measures both syntactic validity (structural correctness) and semantic accuracy (correct value extraction) across varying levels of JSON nesting complexity. Use when the user wants to benchmark on LLMStructBench, or asks about evaluating this task. Reports DOC.

Evaluates generative language models on a suite of multiple-choice and open-ended benchmarks covering reasoning, commonsense, multitask proficiency, and truthfulness. It measures accuracy across diverse domains to assess generalization and the impact of data combination strategies. Use when the user wants to benchmark on AI2 Reasoning Challenge (ARC), HellaSwag, MMLU, TruthfulQA, BigBench, HumanEval, or asks about evaluating this task. Reports accuracy.

Evaluates the impact of multilingual data mixtures on language modeling capability and downstream task performance across multiple languages. It probes whether English dominance or high language count negatively interferes with multilingual model training. Use when the user wants to benchmark on mC4, FineWeb2, or asks about evaluating this task. Reports language modeling loss.

Evaluates on-device meteorological variable forecasting and imputation capabilities using a federated learning framework with personalized adapters. It tests the model's ability to predict regional weather trends and handle missing data under data scarcity and heterogeneous distributions. Use when the user wants to benchmark on On-device Weather Series (ODW1/ODW2), or asks about evaluating this task. Reports MAE.

Evaluates the robustness of label noise learning (LNL) methods on medical image classification tasks. It probes model performance under varying noise types (symmetric, instance-dependent, real-world), noise ratios, and class imbalance distributions across multiple imaging modalities. Use when the user wants to benchmark on PathMNIST, DermaMNIST, BloodMNIST, OrganCMNIST, DRTiD, Kaggle DR+, CheXpert, or asks about evaluating this task. Reports average classification accuracy.

Evaluates machine learning models on two drug discovery tasks: Hit Identification (predicting activity for novel, structurally dissimilar molecules) and Lead Optimization (ranking minor molecular modifications to predict activity changes). It probes a model's ability to generalize to unseen chemical space and capture fine-grained structure-activity relationships. Use when the user wants to benchmark on DRD2-Hi, HIV-Hi, KDR-Hi, Sol-Hi, DRD2-Lo, KCNH2-Lo, KDR-Lo, or asks about evaluating this t...

Evaluates the data loading speed and rendering interactivity of the encube visual analytics framework on a tiled display system under varying data volumes and GPU memory constraints. Use when the user has predictions and gold and needs to compute Load time ($T_{\mathrm{Load}}$).

Evaluates large speech language models on low-resource automatic speech recognition across 25 languages from 9 typologically diverse families. It probes cross-linguistic generalization, script bias (Latin vs. non-Latin), model scaling effects, and the impact of language-aware prompting on transcription accuracy. Use when the user wants to benchmark on LoASR-Bench, or asks about evaluating this task. Reports error rates.

Compares local trajectory planners (DWB and TEB) for mobile manipulators by evaluating path smoothness, end-effector stability, trajectory deviation from a global path, and navigation accuracy/time in static and dynamic simulated environments. Use when the user wants to benchmark on Simulated Environments (Playground, Office, Warehouse), or asks about evaluating this task. Reports $\mathbf{p}_{e}$ (end-effector stability).

This evaluation probes a vision-language model's ability to distinguish in-distribution images from out-of-distribution samples using few-shot prompt tuning. It specifically tests fine-grained regional outlier detection by measuring how well the model separates known classes from diverse OOD datasets and semantically similar near-OOD subsets. Use when the user wants to benchmark on ImageNet-1K & OOD combination, or asks about evaluating this task. Reports FPR95.

Evaluates the ability of neural network architectures (KANs, TKANs, RNNs) to forecast localized weather variables (temperature, precipitation, pressure) one day ahead. Probes nonlinear time-series modeling and regression accuracy under varying data distributions, such as low precipitation versus high temperature variance. Use when the user wants to benchmark on Abidjan, Kigali, or asks about evaluating this task. Reports R².

Evaluates long-context retrieval capabilities on real-world documents where relevant information spans entire texts, such as legal contracts and medical notes. It specifically probes a model's ability to locate and rank relevant passages without relying on truncation or chunking strategies that often bias standard retrievers. Use when the user wants to benchmark on LoCoV1, or asks about evaluating this task. Reports nDCG@10.

This benchmark evaluates the long-term conversational memory of LLM agents by testing their ability to answer questions, summarize events, and generate multi-modal dialogues over very long, multi-session conversations. It probes how well models retain and reason over temporal and causal information across hundreds of turns and thousands of tokens. Use when the user wants to benchmark on LoCoMo, or asks about evaluating this task. Reports F1-score.

This benchmark evaluates long-term conversational memory systems by testing their ability to retrieve relevant dialogue turns and answer questions over extended, multi-session histories. It probes semantic reasoning, temporal tracking, and adversarial robustness across five distinct question categories. Use when the user wants to benchmark on LoCoMo10, or asks about evaluating this task. Reports F1 score.

Evaluates a model's ability to distinguish in-distribution (ID) samples from out-of-distribution (OOD) samples using a threshold-free loss-difference clustering approach. It measures detection performance across standard benchmarks with diverse natural images and hard benchmarks where OOD classes share the same source dataset as ID. Use when the user wants to benchmark on CIFAR100, SVHN, Places, LSUN-Crop, LSUN-Resize, Textures, CIFAR10, TinyImageNet, or asks about evaluating this task. Repor...

Evaluates the classification performance of a parameter-efficient model trained on cached foundation model features with tensor augmentations. It probes the model's ability to generalize across diverse image domains, object categories, and input resolutions using only lightweight classifier heads. Use when the user wants to benchmark on APTOS2019, DDSM, ISIC, AID, NABirds, Flowers102, StanfordCars, StanfordDogs, Oxford-III Pet, Caltech-101, SUN397, or asks about evaluating this task. Reports ...

Compute the log_loss metric — provided by scikit-learn. Use when the user has predictions and ground-truth and needs to compute log_loss, or asks how to score with log_loss.

Evaluates a protein language model's zero-shot capability to score single-site amino acid mutations by comparing contextual likelihoods of wild-type versus mutant residues. It probes how well masked language modeling objectives capture evolutionary and structural constraints for mutant effect prediction. Use when the user has predictions and gold and needs to compute log-odds ratio.

Compute the LogAUC metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute LogAUC, or asks how to score with LogAUC.

Compute the LogCoshError metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute LogCoshError, or asks how to score with LogCoshError.