Claude Skills by qhjqhj00

Evaluates spoken language understanding capabilities across multiple tasks including intent classification, slot filling, emotion recognition, and dialogue act classification. It probes a model's ability to map raw audio inputs to semantic labels, test robustness to noise and low-resource settings, and assess the utility of pretrained ASR/NLU feature extractors in end-to-end speech processing pipelines. Use when the user wants to benchmark on FSC (Fluent Speech Commands), Snips, SLURP, IEMOCA...

Measures how much final code fidelity degrades when a system's design is progressively disclosed through multi-turn interaction rather than provided upfront. It evaluates semantic faithfulness to a committed design and structural integration of dependencies in long-horizon coding agents. Use when the user wants to benchmark on SLUMP benchmark, or asks about evaluating this task. Reports IF50.

Evaluates text-to-query systems across relational, document, and graph database models using four query languages (SQL, MQL, Cypher, SPARQL). It probes the ability of models to translate natural language medical questions into correct, executable database queries using standardized SNOMED-CT aligned synthetic patient data. Use when the user wants to benchmark on SM3-Text-to-Query, or asks about evaluating this task. Reports correctness.

Compute sma2023/wil via the HuggingFace `evaluate` library. Use when the user has predictions + references and wants the canonical implementation of sma2023/wil.

Evaluates large vision-and-language models on mathematical reasoning tasks from the Math Kangaroo Olympiad, testing their ability to solve grade-appropriate (K-12) multiple-choice problems that may require joint text and image interpretation. Use when the user wants to benchmark on SMART-840, or asks about evaluating this task. Reports accuracy.

Evaluates multi-agent reinforcement learning algorithms in a simulated urban driving environment, measuring scenario completion, episode duration, human-like driving fidelity, and traffic rule compliance. Use when the user wants to benchmark on SMARTS (NeurIPS Competition Track-1), or asks about evaluating this task. Reports Completion.

Evaluates neural marked spatio-temporal and temporal point process models on predicting the next event's time, location, and mark, while quantifying prediction uncertainty. It probes the model's ability to generate well-calibrated confidence regions for continuous variables and accurate probability estimates for discrete marks. Use when the user wants to benchmark on Earthquake, Crime, Football, StackOverflow, Retweet, MIMIC-II, Financial Transactions, or asks about evaluating this task. Repo...

Evaluates data efficiency and response generation quality in goal-oriented dialogue systems under few-shot conditions. It measures how well a model can generate contextually appropriate and entity-accurate responses using only a small fraction of in-domain dialogue data. Use when the user wants to benchmark on SMD, or asks about evaluating this task. Reports BLEU.

Evaluates 3D medical image segmentation models on mesoscopic small vessel extraction from ultra-high-resolution (7T) Time-of-Flight Magnetic Resonance Angiography. It probes a model's ability to handle high noise, poor vessel-background contrast, and domain shifts across different MRI acquisition sources. Use when the user wants to benchmark on SMILE-UHURA Challenge Dataset, or asks about evaluating this task. Reports Dice coefficient (DICE).

Evaluates the ability of multimodal large language models (MLLMs) to perform in-context learning (ICL) in medical domains. It probes how effectively models leverage provided image-question-answer demonstrations to answer new clinical queries, while also measuring robustness to irrelevant examples, recency bias, and the gap between automated and expert clinical judgment. Use when the user wants to benchmark on SMMILE, SMMILE++, or asks about evaluating this task. Reports LLM-as-a-Judge.

Evaluates machine translation quality for 115 under-represented languages using professionally translated parallel data. It measures the improvement in character-level n-gram F-score (ChrF) after fine-tuning a baseline model on the Smol dataset compared to the unfine-tuned baseline. Use when the user wants to benchmark on SmolSent, SmolDoc, or asks about evaluating this task. Reports ChrF.

This evaluation probes a vision-language model's ability to perform end-to-end document conversion, including text recognition, layout analysis, table and chart structure extraction, and code/formula parsing. It measures how accurately the model reconstructs document content and spatial structure from page images into standardized markup formats. Use when the user wants to benchmark on DocLayNet, SynthCodeNet, Im2Latex-230k, FinTabNet, PubTables-1M, or asks about evaluating this task. Reports...

Evaluates a vision-language-action model's ability to perform robotic manipulation tasks in both simulated and real-world environments. It probes visuomotor policy generalization, fine-grained task decomposition handling, and the impact of pretraining and inference modes on success rates. Use when the user wants to benchmark on LIBERO, Meta-World, SO100 Real-World Tasks, SO101 Real-World Tasks, or asks about evaluating this task. Reports Success Rate (SR).

Evaluates the ability of physically simulated humanoid agents to perform complex, long-horizon Olympic sports tasks using different control policies and motion priors. It probes task completion accuracy, physical realism, and the effectiveness of adversarial vs. hierarchical reinforcement learning in sparse-reward simulation environments. Use when the user wants to benchmark on SMPLOlympics Sports Environments, or asks about evaluating this task. Reports Suc Rate.

Evaluates Vietnamese social media text classification across four tasks: constructive speech detection, complaint detection, emotion recognition, and hate speech detection. It probes the ability of monolingual versus multilingual BERT-based models to handle low-resource, domain-specific Vietnamese text with varying preprocessing requirements. Use when the user wants to benchmark on VSMEC, ViCTSD, ViOCD, ViHSD, or asks about evaluating this task. Reports macro-average F1 score.

Evaluates the effectiveness of different MCSAT-based bitvector solving strategies and conflict explainers against a standard SMT benchmark suite. It measures how well each solver variant handles fixed-size bitvector formulas under a strict time limit. Use when the user wants to benchmark on SMT-LIB QF_BV, or asks about evaluating this task. Reports solved_instances.

Evaluates the latency reduction, accuracy loss, memory overhead, energy saving, and early exit performance of a semantic memory caching mechanism (SMTM) for accelerating CNN inference on mobile devices. Use when the user wants to benchmark on UCF101, CIFAR-100 (long-tail), or asks about evaluating this task. Reports latency reduction.

Evaluates a single-shot pruning method's ability to identify and remove unimportant network connections at initialization, preserving classification accuracy across varying sparsity levels on standard vision and sequence datasets. Use when the user wants to benchmark on MNIST, CIFAR-10, Tiny-ImageNet, or asks about evaluating this task. Reports accuracy.

Evaluates end-to-end spoken language understanding by measuring how well an embedded system extracts intents and slots from spoken audio. It probes the pipeline's ability to generalize to unseen queries and handle real-world ASR errors under strict resource constraints. Use when the user wants to benchmark on SmartLights, Weather, or asks about evaluating this task. Reports F1.

Evaluates the communication performance and hardware efficiency of Spiking Neural Network-based Decision-Feedback Equalizers (DFEs) for optical channels compared to traditional Artificial Neural Network baselines. It probes the trade-off between bit error rate, computational complexity, and energy efficiency under varying quantization levels and FPGA resource constraints. Use when the user wants to benchmark on Custom Optical Communication DFE Benchmark, or asks about evaluating this task. Re...

Evaluates the classification accuracy of directly-trained spiking neural networks (SNNs) on both static image recognition and neuromorphic event-based vision tasks. It probes the model's ability to maintain gradient stability and high predictive performance while operating with minimal simulation timesteps, highlighting efficiency gains over traditional ANN-SNN conversion methods. Use when the user wants to benchmark on CIFAR-10, ImageNet, DVS-Gesture, DVS-CIFAR10, or asks about evaluating th...

Evaluates the robustness of audio deepfake detection models under varying signal-to-noise ratios (SNRs) by testing binary (real vs. spoof) and four-class (real+clean, real+noisy, spoof+clean, spoof+noisy) classification tasks on ASVspoof 2021 utterances augmented with MS-SNSD ambient noise. Use when the user wants to benchmark on ASVspoof 2021 (DF), or asks about evaluating this task. Reports EER.

Evaluates the detection capability of the Lunar Gravitational-Wave Antenna (LGWA) for massive binary black hole mergers by computing the signal-to-noise ratio (SNR) of observed and simulated events against fixed thresholds. Use when the user has predictions and gold and needs to compute SNR.

Evaluates the ability of audio and text models to generate clinically accurate, well-structured SOAP notes from long-form doctor-patient conversations. Probes long-context audio reasoning, fact-grounding, and clinical documentation quality. Use when the user wants to benchmark on Doctor-Patient SOAP Conversations, or asks about evaluating this task. Reports Faithfulness.

Evaluates the hallucination rate of LLM-generated medical SOAP notes against physician-patient transcripts. It compares a literal, inference-unaware evaluation framework against a clinically informed, inference-aware framework to measure how often valid clinical reasoning is incorrectly flagged as hallucination. Use when the user wants to benchmark on Physician-Patient Transcripts, or asks about evaluating this task. Reports Mean Hallucination Rate.

Evaluates large language models on zero-shot and chain-of-thought cell type annotation tasks using single-cell RNA-seq gene expression profiles. It probes the models' ability to translate structured genomic data into textual descriptions and accurately predict cell type labels without fine-tuning. Use when the user wants to benchmark on SOAR-RNA, or asks about evaluating this task. Reports Average BLEU.

Evaluates a learning-based algorithm selection framework for High-Level Synthesis Design Space Exploration (DSE). It measures how accurately the model recommends the best-performing DSE algorithm for a given benchmark, and assesses the resulting optimization performance (ADRS) and runtime compared to heuristic and reinforcement learning baselines. Use when the user wants to benchmark on MachSuite & Polyhedral Benchmarks, or asks about evaluating this task. Reports recommendation_accuracy.

Evaluates the energy efficiency, throughput, and output quality of a custom edge server built from 60 mobile SoCs against traditional CPU and GPU servers for video transcoding workloads. Use when the user wants to benchmark on vbench, or asks about evaluating this task. Reports streams/W.

This evaluation protocol assesses a model's capability to predict binary drug-target interactions (DTI) using graph-based representations. It specifically probes performance under both balanced and highly imbalanced data distributions, as well as generalization to unseen drugs or targets in cold-start scenarios. Use when the user wants to benchmark on KIBA, Davis, BindingDB, DrugBank, or asks about evaluating this task. Reports AUROC.

Evaluates multimodal video-language models on soccer-specific tasks: referee decision validation via question-answering and multi-label action classification. It probes the model's ability to align visual, auditory, and textual cues with ground-truth soccer events and rules. Use when the user wants to benchmark on XFoul validation dataset, SoccerNet-v2, or asks about evaluating this task. Reports QwQ Scorer, F1 Score (wt).

This evaluation probes an LLM's ability to predict individual human responses in social science experiments and match the overall distribution of those responses. It measures both point-wise accuracy and distributional alignment across unseen studies, conditions, outcomes, and participant demographics. Use when the user wants to benchmark on SocSci210, or asks about evaluating this task. Reports Accuracy, Wasserstein distance.

Evaluates a model's ability to reason about and generate rules-of-thumb (RoTs) that capture social and moral norms across 12 distinct dimensions of judgment, such as cultural pressure, legality, and moral foundations. It probes whether neural models can produce attribute-aware, context-sensitive normative judgments for unseen social scenarios. Use when the user wants to benchmark on Social-Chem-101, or asks about evaluating this task. Reports micro-F1.

This benchmark evaluates the ability of models to automatically detect multiple dimensions of media bias (e.g., hate speech, racial, gender, political, linguistic, and text-level context bias) in social media posts across different topic domains. It probes a model's robustness to domain shift and severe class imbalance in multi-label bias identification tasks. Use when the user wants to benchmark on Social Media Bias Dataset (YouTube & Reddit), or asks about evaluating this task. Reports weig...

Probes intersectional social bias in Large Vision-Language Models by measuring how model outputs vary when only perceived race, gender, or physical attributes change in counterfactual images. It specifically evaluates toxicity, stereotypical language, and competency ratings across different demographic groups. Use when the user wants to benchmark on SocialCounterfactuals, or asks about evaluating this task. Reports MaxToxicity.

Evaluates Vision-Language Models' ability to perform spatial, spatiotemporal, and social reasoning in dynamic, crowd-filled robot navigation scenarios. It probes scene understanding by asking VLMs to answer visual questions based on image sequences and Bird's Eye View (BEV) representations. Use when the user wants to benchmark on SocialNav-SUB, or asks about evaluating this task. Reports PA.

This benchmark evaluates a model's ability to recognize coordinated group activities in soccer matches by comparing two input modalities: raw video pixels and structured positional tracking data. It probes spatial-temporal reasoning, tactical formation understanding, and robustness to visual shifts by measuring how well models classify 10 distinct group actions from synchronized match footage. Use when the user wants to benchmark on SoccerNet-GAR, or asks about evaluating this task. Reports b...

Evaluates neural models on three information extraction sub-tasks in materials science: detecting experiment-describing sentences, extracting and typing entity mentions (materials, values, devices), and filling experiment-specific slots (e.g., temperature, anode material). Use when the user wants to benchmark on SOFC-Exp Corpus, Synthesis Procedures Dataset, or asks about evaluating this task. Reports macro-average F1.

Evaluates the reliability and discriminative power of automatic machine translation metrics by comparing their statistical significance against human MQM judgments. It measures how well a metric's pairwise system rankings align with human preferences using permutation-based p-values rather than hard binary decisions. Use when the user has predictions and gold and needs to compute Soft Pairwise Accuracy (SPA).

Evaluates a diffusion-based molecular language model's ability to generate chemically valid, drug-like molecules and optimize them for specific protein targets. It probes distribution matching, structural diversity, and target-aware binding affinity prediction. Use when the user wants to benchmark on ZINC-Curated, SMILES, SAFE, or asks about evaluating this task. Reports Novel Top-hit 5% Score.

Evaluates pretrained language models' in-distribution (ID) and out-of-distribution (OOD) classification accuracy under single-setting and multi-setting fine-tuning configurations. It probes how training dynamics and subset selection affect robustness and generalization across languages, sources, and tasks. Use when the user wants to benchmark on SST-2, IMDB, Yelp, Sentiment140, RTE, QQP, or asks about evaluating this task. Reports accuracy.

Evaluates the ability of multilayer perceptrons to predict vegetation phenology parameters (start of season, peak of season, peak NDVI value) from soil temperature and meteorological variables in subarctic grasslands. It probes how well non-linear models capture complex, non-linear interactions between climate drivers and vegetation dynamics compared to simple linear baselines. Use when the user wants to benchmark on Subarctic grassland phenology dataset (Iceland, 2014-2019), or asks about ev...

Evaluates tree-based machine learning models for day-ahead solar power generation forecasting at hourly resolution. It probes the models' ability to capture spatial heterogeneity in meteorological features and their robustness under varying weather conditions. Use when the user wants to benchmark on Belgian Solar Power Generation Dataset, or asks about evaluating this task. Reports RMSE.

Evaluates the predictive accuracy of ensemble machine learning models for forecasting solar power generation using meteorological parameters. It probes regression performance under varying feature sets and ensemble aggregation strategies. Use when the user wants to benchmark on SRRA dataset, or asks about evaluating this task. Reports RMSE.

This benchmark evaluates the capability of machine learning models to detect offensive language in Sinhala text. It probes binary text classification performance on a highly imbalanced dataset of Sinhala tweets, measuring how well models distinguish between offensive and non-offensive content. Use when the user wants to benchmark on SOLD, or asks about evaluating this task. Reports macro-averaged F1-score.

This benchmark evaluates a model's ability to perform open-vocabulary 3D instance segmentation on indoor point clouds. It probes the model's capacity to align 3D geometric features with free-form language instructions to generate accurate instance masks for both seen and unseen categories. Use when the user wants to benchmark on ScanNetv2, ScanNet200, Replica, or asks about evaluating this task. Reports AP.

Evaluates the ability of token classification models to identify and categorize software mentions within academic sentences. It probes how well models handle class imbalance, subtoken segmentation, and syntactic complexity in scholarly text. Use when the user wants to benchmark on SOMD (Software Mention Detection in Scholarly Publications), or asks about evaluating this task. Reports F1-Score.

Probes the capability of joint entity and relation extraction for identifying software mentions and their attributes (URLs, versions, licenses) in scholarly articles. It specifically tests in-distribution performance and out-of-distribution generalization across two competition phases. Use when the user wants to benchmark on SOMD 2025, or asks about evaluating this task. Reports F1-score.

Evaluates the accuracy and spatial-temporal fidelity of a machine learning-derived daily soil moisture product for Europe. It probes the model's ability to generalize across diverse climates, capture drought dynamics, and outperform existing reanalysis and satellite-based soil moisture datasets. Use when the user wants to benchmark on SoMo.ml-EU, or asks about evaluating this task. Reports uRMSD.

Evaluates a model's ability to extrapolate daily soil moisture dynamics across three depth layers using in-situ ground measurements and meteorological forcing. It probes temporal fidelity and absolute accuracy against independent station data. Use when the user wants to benchmark on ISMN & CEMADEN in-situ soil moisture measurements, or asks about evaluating this task. Reports NRMSE.

Evaluates the effectiveness of an adversarial perturbation method (SongBsAb) designed to prevent illegal singing voice conversion. It probes the method's ability to disrupt singer identity and lyrical fidelity in converted audio while maintaining high audio quality and imperceptibility. Use when the user wants to benchmark on OpenSinger, NUS-48E, or asks about evaluating this task. Reports Lyric Word Error Rate (WER).