Claude Skills by qhjqhj00

Evaluates contextual code embeddings on six Python source-code understanding tasks, including classification of variable misuse, incorrect binary operators, swapped operands, function-docstring mismatches, and exception types, plus a joint localization and repair task. Use when the user wants to benchmark on ETH Py150 Open Benchmarks, or asks about evaluating this task. Reports classification accuracy.

This evaluation probes the per-evidence-item utility and trace sensitivity in single-shot retrieval-augmented generation. It measures how removing, replacing, or duplicating retrieved context chunks affects answer correctness, grounding faithfulness, confidence calibration, and reasoning trace stability. Use when the user wants to benchmark on HotpotQA (distractor setting), 2WikiMultihopQA, or asks about evaluating this task. Reports Soft Correctness.

Evaluates Chinese language understanding and generation capabilities across a hierarchical framework. It probes discourse comprehension, conversational interaction, mathematical reasoning, and multilingual tasks using a multi-level scoring strategy that normalizes model performance against a fixed baseline. Use when the user wants to benchmark on CUGE (lite version), or asks about evaluating this task. Reports normalized capability performance.

Evaluates lane detection performance in diverse urban and highway scenarios using an F1-measure based on IoU between predicted and ground truth lane lines. Use when the user wants to benchmark on CULane, or asks about evaluating this task. Reports F1-measure.

Probes LLMs' cross-cultural emotion understanding by testing their ability to predict emotions and sentiments across six languages, specifically examining how prompt language and explicit country context influence model performance. Use when the user wants to benchmark on CULEMO, or asks about evaluating this task. Reports emotion prediction.

This benchmark probes a machine translation model's ability to accurately translate culture-loaded expressions (idioms, proverbs, culture-specific items) while preserving their figurative, contextual, and cultural meanings. It evaluates whether models can avoid literal or superficial translations that strip away culturally grounded nuances. Use when the user wants to benchmark on CulT-Eval, or asks about evaluating this task. Reports ACRE.

Evaluates machine translation systems on culturally specific items (CSIs) to measure how well they preserve cultural nuances, entities, and meanings compared to reference translations. It probes both automated lexical/semantic alignment and human judgment on translation accuracy for culturally grounded content. Use when the user wants to benchmark on Wikipedia Cultural Parallel Corpus, or asks about evaluating this task. Reports CSI-Match.

Assesses the ability of large multimodal models to identify the geographical origin (country, subregion, or continent) of an image based on visual cultural cues. It probes implicit stereotypical associations and geographic bias in vision-language models. Use when the user wants to benchmark on Dalle Street, Dollar Street, MaRVL, or asks about evaluating this task. Reports classification accuracy.

This benchmark evaluates how well multilingual LLMs preserve cultural nuance, idioms, puns, and culturally embedded concepts during machine translation. It probes the persistent gap between grammatical accuracy and cultural resonance by measuring translation quality across different figurative and non-figurative segment categories. Use when the user wants to benchmark on Cultural Nuance MT Benchmark, or asks about evaluating this task. Reports overall quality.

Evaluates whether an LLM's value profile aligns with specific cultural norms using World Values Survey data, and tests the model's steerability when provided with diverse cultural contexts. It probes the extent to which constitutional AI codifies dominant cultural biases and resists prompt-based cultural adaptation. Use when the user wants to benchmark on World Values Survey (WVS) Wave 7, or asks about evaluating this task. Reports Pearson correlation.

Evaluates multilingual content safety guard models on their ability to detect harmful or unsafe prompts and responses across diverse languages and cultural contexts, including zero-shot generalization to unseen languages. Use when the user wants to benchmark on CultureGuard, PolyGuardPrompts, RTP-LX, MultiJail, XSafety, Aya Red-teaming, or asks about evaluating this task. Reports harmful-F1.

Evaluates piecewise-stationary multi-armed bandit algorithms by measuring the expected cumulative regret over a sequence of time steps. It probes how well an algorithm adapts to changing arm reward distributions (change-points) while balancing exploration and exploitation. Use when the user has predictions and gold and needs to compute cumulative regret.

Evaluates machine translation quality for Czech-Ukrainian and Ukrainian-Czech translation using constrained back-translation systems and a proprietary unconstrained system. Probes the impact of data preprocessing techniques like romanization and ensemble methods on translation performance. Use when the user wants to benchmark on Flores 101 development set, WMT22 Czech-Ukrainian test set, or asks about evaluating this task. Reports BLEU.

Probes a clinical language model's ability to predict binary adverse outcomes from free-text EHR notes while simultaneously calibrating its prediction uncertainty. It evaluates both discriminative accuracy and probabilistic calibration across multiple clinical risk stratification tasks. Use when the user wants to benchmark on MIMIC-IV, or asks about evaluating this task. Reports AUROC.

Evaluates conversational AI assistants' ability to maintain user-specific awareness and correctly prioritize safety-critical constraints over conflicting preferences in multi-turn interactions. It probes whether models can distinguish hard safety limits from softer user desires and avoid generic or evasive responses. Use when the user wants to benchmark on CURATe, or asks about evaluating this task. Reports pass rates.

This benchmark evaluates multimodal large language models' ability to perform clinical differential diagnosis using patient history and medical images. It explicitly disentangles intrinsic diagnostic reasoning from external evidence retrieval by testing models under four paradigms: no context, physician-curated references, standard RAG, and agentic web search. The protocol probes how well models leverage retrieved literature versus relying on internal knowledge, and how retrieval noise impact...

Evaluates automated red-teaming methods on their ability to generate diverse and effective prompts that elicit toxic responses from target LLMs. It probes both the effectiveness (toxicity elicitation rate) and diversity (textual and semantic variation) of generated test cases across text continuation and instruction-following tasks. Use when the user wants to benchmark on IMDb review dataset, Alpaca dataset, Databricks dataset, or asks about evaluating this task. Reports toxic response rate.

Evaluates continual learning capabilities in language models by measuring skill retention, forward/backward transfer, and catastrophic forgetting across a developmental skill graph spanning ages 5–10. It probes how sequential, joint, and independent training affect performance on instruction, context-question-answer, and context-sentence-question-answer tasks. Use when the user wants to benchmark on CurLL, or asks about evaluating this task. Reports LLM rating score (1-5).

Tests a model's ability to learn sequentially across multiple NLP tasks while retaining prior knowledge. It specifically probes catastrophic forgetting mitigation during continual fine-tuning by measuring performance drops on earlier tasks after learning new ones. Use when the user wants to benchmark on GLUE-MRPC, GLUE-SST-2, Sentiment140, WikiText-2, or asks about evaluating this task. Reports Accuracy.

Evaluates the out-of-domain generalization and reasoning capabilities of vision-language models across visual detection, classification, and multimodal mathematical reasoning tasks, alongside standard multimodal benchmarks. Use when the user wants to benchmark on RefCOCO, RefGTA, Pascal-VOC, Math360K, CLEVER-70k-Counting, MathVista, MATH, AI2D, MMBench, MMVet, OCRBench, LLaVABench, or asks about evaluating this task. Reports accuracy.

Evaluates text-to-image generation models on their ability to align generated images with text prompts, produce visually appealing outputs, and match human preferences. It tests the effectiveness of curriculum-based fine-tuning strategies on standard generative benchmarks. Use when the user wants to benchmark on D1 (Black-ICLR-2024), D2 (DrawBench), D3 (Pick-a-Pic), or asks about evaluating this task. Reports Text Alignment.

Evaluates how the ordering of training data (curriculum) affects the quality of task-specific word embeddings. It probes whether optimized data sequencing improves downstream performance in sentiment analysis, NER, POS tagging, and parsing compared to random or heuristic orderings. Use when the user wants to benchmark on Wikipedia Paragraph Corpus, or asks about evaluating this task. Reports test results.

Evaluates a kernel-level safety gateway's ability to correctly classify MCP tool-call prompts as dangerous or benign across 18 attack and benign domains. It probes the system's semantic understanding of tool intent versus surface-form rule matching, measuring how well the logit-based safety primitive prevents privilege escalation and adversarial bypasses. Use when the user wants to benchmark on Custom-101, or asks about evaluating this task. Reports F1.

Evaluates end-to-end and cascaded named entity recognition from Arabic speech. It probes a model's ability to jointly transcribe spoken Arabic and predict fine-grained entity types (21 categories) using BIO-style tagging, as well as extract entity spans and values. Use when the user wants to benchmark on CV-18 NER, or asks about evaluating this task. Reports CoER.

Evaluates end-to-end inference latency and hardware efficiency of computer vision models on edge AI hardware. It probes how well a hardware-software co-design optimizes data movement and compute utilization under strict memory and bandwidth constraints. Use when the user wants to benchmark on ImageNet, COCO 2017, or asks about evaluating this task. Reports Latency [ms].

Evaluates how well large language models align with culturally grounded Chinese value rules compared to Western benchmarks. It probes moral reasoning, preference alignment, and boundary separation across six sensitive themes like drugs, firearms, politics, and suicide. Use when the user wants to benchmark on CVC, or asks about evaluating this task. Reports preference.

This benchmark evaluates the cultural and linguistic understanding of multimodal vision-language models by testing their ability to answer multiple-choice questions about images in diverse languages and cultural contexts. It probes zero-shot generalization across location-aware and location-agnostic prompts, highlighting performance gaps in low-resource languages. Use when the user wants to benchmark on CVQA, or asks about evaluating this task. Reports accuracy.

Evaluates the ranking performance of models for click-through rate (CTR) and post-click conversion rate (CVR) estimation in recommendation systems. It probes the model's ability to correctly rank items by their predicted probability of conversion, while mitigating sample selection bias and false independence assumptions between clicks and conversions. Use when the user wants to benchmark on Industrial Benchmark, Ali-CCP, or asks about evaluating this task. Reports AUC.

Evaluates the quality of novel view synthesis from sparse input views using 3D radiance fields. It measures how well the model reconstructs unseen images and maintains 3D consistency across different sparsity levels (3, 6, or 9 input views). Use when the user wants to benchmark on DTU dataset, Synthetic dataset, or asks about evaluating this task. Reports PSNR.

Evaluates whether LLM-generated code is simultaneously functionally correct and secure against vulnerabilities. It measures the pass rate for functionality alone versus the joint pass rate for functionality and security, highlighting the gap where models produce working but vulnerable code. Use when the user wants to benchmark on CWEVAL-BENCH, or asks about evaluating this task. Reports func-sec@k.

This benchmark evaluates a model's ability to perform binary classification on lexical complexity, determining whether a given word is perceived as complex or non-complex by human readers. Use when the user wants to benchmark on SemEval CWI, or asks about evaluating this task. Reports F1 score.

Evaluates multimodal large language models on chest X-ray diagnosis across visual understanding, text generation, spatiotemporal alignment, and foundational medical language capabilities. It probes the model's ability to interpret radiological images, generate clinical reports, localize anomalies, and reason over medical text. Use when the user wants to benchmark on MIMIC-CXR & CheXpert, OpenI, Language Datasets (CHIP-CDN, CMeEE, IMCS-V2-MRG, DDx-basic, MedSafety, MedHG, Med-Exam), MS-CXR, RS...

This benchmark evaluates the capability of vision-language models to generate accurate and clinically relevant free-text radiology reports from chest X-ray images. It probes both linguistic quality through standard NLG metrics and diagnostic accuracy by extracting and comparing clinical abnormality labels against ground truth reports. Use when the user wants to benchmark on IU X-ray, MIMIC-CXR, CheXpert Plus, or asks about evaluating this task. Reports CIDEr.

Evaluates the capability of object detection models to localize thoracic abnormalities in chest X-rays under a weakly semi-supervised setting. It specifically probes how well models can leverage sparse point-level annotations alongside a small fraction of fully bounding-box-labeled images to achieve accurate region detection. Use when the user wants to benchmark on RSNA, VinDr-CXR, or asks about evaluating this task. Reports mAP.

Evaluates multi-stage structured diagnostic reasoning in chest X-rays, probing a model’s ability to perform visual grounding, anatomical segmentation, quantitative measurement derivation, and clinical threshold application. It tests whether models can consistently link abstract diagnostic criteria with accurate visual interpretation across direct and guided reasoning paths. Use when the user wants to benchmark on CXReasonBench, or asks about evaluating this task. Reports Completion.

Evaluates multi-turn diagnostic reasoning agents on chest X-rays, measuring their ability to identify tasks, extract evidence, maintain coverage, avoid hallucinations, and sustain coherent dialogue success. Use when the user wants to benchmark on CXReasonDial, or asks about evaluating this task. Reports Faithfulness (Faith).

Evaluates robust multi-label classification under long-tailed class distributions and open-world zero-shot generalization to unseen rare diseases in chest X-rays. Use when the user wants to benchmark on PadChest + NIH, or asks about evaluating this task. Reports mAP.

Evaluates language models' cybersecurity capabilities by testing their ability to solve real-world Capture the Flag (CTF) challenges in an agent-based environment. It probes iterative problem-solving, command execution in a Linux container, and vulnerability exploitation under constrained iteration and token limits. Use when the user wants to benchmark on Cybench, or asks about evaluating this task. Reports Unguided Performance.

Evaluates large language models' knowledge of cybersecurity certifications across a spectrum from general IT security to specialized operational technology (OT) and vendor-specific procedural knowledge. It probes whether models can meet professional certification passing standards and identifies gaps in formal, safety-critical industrial protocols. Use when the user wants to benchmark on CyberCertBench, or asks about evaluating this task. Reports accuracy.

This evaluation probes the impact of LLM assistance on human cybersecurity practitioners' ability to execute novel cyberattack challenges. It measures objective performance metrics (phase completion rates and time) and subjective perception (sentiment/mental effort) across inexperienced and highly skilled cohorts, comparing LLM-assisted versus unassisted conditions. Use when the user wants to benchmark on CYBERSECEVAL 3 Challenge Set, or asks about evaluating this task. Reports phase completi...

Evaluates the ability of generative models to perform unpaired image-to-image translation while preserving structural integrity and achieving perceptual realism. Probes domain mapping capabilities without requiring paired training data. Use when the user wants to benchmark on Cityscapes, Google Maps aerial photos & maps, or asks about evaluating this task. Reports FCN score.

Evaluates a model's ability to generate scene graphs from aerial video sequences by predicting object relationships and interactions. It specifically probes long-range temporal dependency modeling and periodic interaction recognition in drone-captured footage across predicate classification, scene graph classification, and scene graph detection tasks. Use when the user wants to benchmark on AeroEye, PVSG, ASPIRe, or asks about evaluating this task. Reports mean Recall@K (mR@K).

This benchmark evaluates the computational performance and scaling behavior of the cygrid gridding module. It measures processing time and parallelization efficiency across varying input sample sizes, field dimensions, and core counts. Use when the user has predictions and gold and needs to compute processing_time.

Evaluates LLMs' ability to generate precise Cypher queries from natural language questions over large-scale property graphs. It probes complex graph retrieval capabilities including multi-hop reasoning, temporal constraints, aggregations, and strict schema adherence. Use when the user wants to benchmark on CypherBench, or asks about evaluating this task. Reports EX.

Evaluates NLP models on four challenging text classification tasks using a large-scale Czech news dataset: identifying the news source, article category, inferred author gender, and publication day of the week. The benchmark tests a model's ability to capture deep textual dependencies and contextual cues beyond simple keyword matching. Use when the user wants to benchmark on CZE-NEC, or asks about evaluating this task. Reports F1 Macro.

Compute CZLC/rouge_raw via the HuggingFace `evaluate` library. Use when the user has predictions + references and wants the canonical implementation of CZLC/rouge_raw.

Compute d-matrix/dmx_perplexity via the HuggingFace `evaluate` library. Use when the user has predictions + references and wants the canonical implementation of d-matrix/dmx_perplexity.

Compute d-matrix/dmxMetric via the HuggingFace `evaluate` library. Use when the user has predictions + references and wants the canonical implementation of d-matrix/dmxMetric.

Evaluates a model's ability to detect and quantify the degree of replication between an original image and a diffusion-generated replica. It probes continuous replication level prediction rather than binary copy detection, measuring how well predicted scores align with manually annotated replication levels. Use when the user wants to benchmark on D-Rep, or asks about evaluating this task. Reports PCC.

Evaluates LLMs' vulnerability to deceptive reasoning and jailbreak attacks by measuring how well models align their internal chain-of-thought with malicious instructions while producing benign final outputs. It probes detection evasion, output camouflage, and internal malicious reasoning under adversarial system prompt injections. Use when the user wants to benchmark on D-REX, or asks about evaluating this task. Reports Target-Specific Success (%).