Claude Skills by qhjqhj00

Evaluates machine learning surrogates for 2D airfoil aerodynamics on prediction accuracy, computational speed-up, physical consistency, and out-of-distribution generalization. The benchmark compares learned models against a standard CFD solver (OpenFOAM) across in-distribution and novel geometric configurations. Use when the user wants to benchmark on ML4CFD Competition Dataset, or asks about evaluating this task. Reports Global Score.

Evaluates the cross-dataset generalization capability of voice anti-spoofing models. It probes whether models trained on one synthetic audio dataset can accurately detect deepfake or spoofed speech when tested on entirely different datasets, including those with only spoof samples or different languages. Use when the user wants to benchmark on ASVspoof19, ASVspoof21-DF, ASVspoof21-LA, FakeOrReal, InTheWild, MLAAD v1, Voc.v, WaveFake, or asks about evaluating this task. Reports accuracy.

Evaluates retrieval over long multilingual documents (up to 8,192 tokens), testing a model's ability to capture information from extended contexts across multiple languages. Use when the user wants to benchmark on MLDR, or asks about evaluating this task. Reports nDCG@10.

Evaluates an agent's ability to iteratively refine and improve runnable solutions for competition-style ML tasks over a long horizon. It probes sustained experiment improvement and competitive performance rather than just initial submission validity. Use when the user wants to benchmark on MLE-Bench Lite, or asks about evaluating this task. Reports Any Medal%.

This benchmark evaluates the capability of embedding models to perform legal information retrieval across diverse jurisdictions, document types, and legal tasks. It probes how well models understand judicial reasoning, regulatory interpretation, and multinational contract analysis compared to general-purpose IR models. Use when the user wants to benchmark on MLEB, or asks about evaluating this task. Reports NDCG@10.

Evaluates LLM agents on open-ended AI research tasks across 13 diverse benchmarks. It measures the agent's ability to navigate codebases, run experiments, and improve model performance, assessing capabilities from reproducing existing research to achieving state-of-the-art results. Use when the user wants to benchmark on MLGym Benchmarks, or asks about evaluating this task. Reports AutoML-inspired optimization metric.

This benchmark evaluates a model's ability to perform simple sequential reasoning tasks (e.g., counting, copying, list intersection) while extrapolating to longer input sequences. It specifically probes length generalization and the impact of multilingual in-context examples on reasoning robustness across different languages. Use when the user wants to benchmark on MLissard, or asks about evaluating this task. Reports accuracy.

Evaluates continual learning capabilities in multimodal large language models across two settings: Domain Continual Learning (DCL) for acquiring IID domain-specific knowledge (remote sensing, medical, autonomous driving, science, finance) and Ability Continual Learning (ACL) for non-IID skill acquisition (OCR, math & logic, visual perception, GUI agent). It measures the model's ability to learn new tasks sequentially while mitigating catastrophic forgetting of previously learned capabilities....

Evaluates the ability of multimodal large language models (MLLMs) to generate accurate image descriptions and answer questions without hallucinating non-existent objects or attributes. It probes object detection, attribute recognition, and spatial understanding under various prompts. Use when the user wants to benchmark on CHAIR (MSCOCO subset), POPE (COCO subset), MME (Hallucination subset), MMBench, or asks about evaluating this task. Reports CHAIR_s.

Evaluates a model's ability to predict the next item in a user's sequential interaction history. It probes the model's capacity to capture temporal dependencies, cross-channel correlations in item embeddings, and cross-feature interactions using only MLP-based architectures. Use when the user wants to benchmark on MovieLens-100k, Amazon Beauty, or asks about evaluating this task. Reports NDCG@10.

Evaluates the inference accuracy of deep neural networks when simulated with Adaptive Block Floating-Point (ABFP) number representation and analog-to-digital converter (ADC) noise. It probes how tile width, amplification gain, and bitwidth affect model quality, and compares the effectiveness of Quantization-Aware Training (QAT) versus Differential Noise Finetuning (DNF) in recovering baseline float32 performance. Use when the user wants to benchmark on MLPerf datacenter inference benchmark, o...

Evaluates real-time perception capabilities for automotive systems, specifically 2D object detection, 2D semantic segmentation, and 3D object detection. It measures how well inference engines meet strict latency constraints and accuracy tolerances required for safety-critical driving tasks. Use when the user wants to benchmark on nuScenes, MLCommons Cognata Dataset, or asks about evaluating this task. Reports tail_latency.

Evaluates the distributed training efficiency of the BERT-Large model on variable-length masked language modeling tasks. It measures how quickly the model converges to a target accuracy and the sustained throughput achieved across multiple GPUs. Use when the user wants to benchmark on MLPerf BERT, or asks about evaluating this task. Reports Time to 72% MLM accuracy.

This protocol evaluates the inference performance of DNN models under various quantization precisions (FP16/INT8, static/dynamic) across multiple inference frameworks on edge and server hardware. It measures standard accuracy and latency metrics using the MLPerf Edge Inference benchmark suite. Use when the user wants to benchmark on ImageNet ILSVRC2012, or asks about evaluating this task. Reports accuracy.

Evaluates the performance and energy efficiency of a proposed deep learning hardware architecture against state-of-the-art GPUs and specialized accelerators. It probes the architecture's ability to handle diverse DL workloads (CNNs, transformers, RNNs) across different batch sizes and its software maturity for general-purpose mapping. Use when the user wants to benchmark on MLPerf benchmark suite, or asks about evaluating this task. Reports Speedup, Rel. Efficiency.

Evaluates end-to-end performance of HPC systems for scientific machine learning, focusing on data staging, I/O efficiency, and model convergence under massive dataset constraints. It measures how well systems handle large-scale volumetric and high-resolution image workloads while meeting strict accuracy targets. Use when the user wants to benchmark on CosmoFlow, DeepCAM, or asks about evaluating this task. Reports MAE, IOU.

Evaluates ML inference systems across diverse hardware and software stacks under realistic deployment scenarios. It measures both model quality against strict baselines and system performance (latency/throughput) to enable architecture-neutral comparisons of production-like workloads. Use when the user wants to benchmark on ImageNet, COCO, WMT16 EN-DE, or asks about evaluating this task. Reports Top-1 accuracy.

This evaluation protocol measures the throughput and latency of machine learning input data pipelines across standard computer vision and NLP benchmarks. It probes how efficiently a data processing framework can ingest, transform, and feed batches to a training loop compared to sequential baselines and competing systems. Use when the user wants to benchmark on ImageNet, COCO, WMT16, WMT17, or asks about evaluating this task. Reports epoch duration.

Evaluates mobile AI inference performance across computer vision and NLP tasks on resource-constrained hardware. It measures both model accuracy against strict thresholds and hardware/software stack efficiency under realistic deployment conditions. Use when the user wants to benchmark on ImageNet 2012 validation, COCO 2017 validation, ADE20K validation, SQuAD v1.1 Dev, or asks about evaluating this task. Reports Top-1 accuracy.

Evaluates the energy efficiency of machine learning systems across diverse hardware scales (data center, edge, tiny) and workloads (inference and training). It measures how effectively systems convert electrical energy into computational progress, tracking improvements in samples processed per joule over time and across system configurations. Use when the user wants to benchmark on MLPerf, or asks about evaluating this task. Reports samples per joule.

Evaluates the classification accuracy and hardware efficiency of CNN accelerators on resource-constrained TinyML workloads. It probes the trade-off between inference latency, energy consumption, and model accuracy under post-training approximate matrix decomposition. Use when the user wants to benchmark on MLPerfTiny, or asks about evaluating this task. Reports Top-1 Accuracy.

Evaluates whether differentiable NAS methods can discover neural architectures that maximize classification accuracy while minimizing energy consumption and latency. The protocol enforces strict weight memory constraints on edge hardware and measures real-world deployment metrics on the NUCLEO-H743ZI2 MCU. Use when the user wants to benchmark on Image Classification (CIFAR-10), Visual Wake Word (MSCOCO 2014), KeyWord Spotting (Speech Commands v2), or asks about evaluating this task. Reports a...

Evaluates the performance and communication overhead of fault-tolerant 2-D allreduce algorithms compared to standard allreduce during data-parallel ML training on TPU-v3 mesh networks. It measures end-to-end training latency and relative efficiency under simulated chip failure conditions. Use when the user wants to benchmark on MLPerf-v0.7 ResNet-50, MLPerf-v0.7 BERT, or asks about evaluating this task. Reports Relative Efficiency.

Evaluates the training performance and scalability of machine learning implementations across diverse hardware and software stacks by measuring time to solution under standardized model and hyperparameter configurations. Use when the user wants to benchmark on MLPerf Training Suite, or asks about evaluating this task. Reports time to solution.

Evaluates a graph neural network's ability to reconstruct particle-flow objects (charged and neutral hadrons) from detector-level tracks and calorimeter clusters in high-pileup simulated events. It probes multi-task learning for particle classification and momentum/energy regression under realistic collider conditions. Use when the user wants to benchmark on DELPHES simulated QCD multijet and ttbar events, or asks about evaluating this task. Reports Efficiency.

Evaluates cross-lingual extractive question answering by measuring how well models can answer questions in one language using context in another, and how performance generalizes across different language pairs. Use when the user wants to benchmark on MLQA, or asks about evaluating this task. Reports F1 score.

This benchmark evaluates a model's ability to perform cross-lingual extractive reading comprehension in a zero-shot setting. It probes how well a model trained on high-resource English (and Chinese) data can generalize to answer questions in low-resource languages by leveraging translated parallel corpora and multilingual attention mechanisms. Use when the user wants to benchmark on MLQA, XQuAD, or asks about evaluating this task. Reports F1.

Evaluates automatic speech recognition (ASR) systems on multilingual read speech data. It measures word error rate across multiple languages and decoding strategies to assess model performance and data scaling effects. Use when the user wants to benchmark on MLS, or asks about evaluating this task. Reports WER.

Evaluates abstractive text summarization models across multiple languages (French, German, Spanish, Russian, Turkish) to measure generation quality and investigate cross-lingual performance gaps and model biases. Use when the user wants to benchmark on MLSUM, or asks about evaluating this task. Reports ROUGE-L.

Evaluates long-horizon video understanding and reasoning capabilities of multimodal models on extended video sequences. Use when the user wants to benchmark on MLVU, or asks about evaluating this task. Reports M-Avg.

Evaluates how well multi-modal large language models align with human preferences when answering open-ended questions about diverse images. It probes the model's ability to follow complex instructions, handle real-world scenarios, and produce responses that match human expectations better than baseline models. Use when the user wants to benchmark on MM-AlignBench, or asks about evaluating this task. Reports Win Rate.

This benchmark evaluates reasoning-intensive retrieval capabilities across text-only and multimodal settings. It probes models' ability to align visual and textual information, navigate technical domain queries, and rank relevant documents or images based on complex, multi-modal prompts. Use when the user wants to benchmark on MM-BRIGHT, or asks about evaluating this task. Reports nDCG@10.

Evaluates the predictive capability of vision-language models on food-related tasks, specifically testing their ability to estimate nutritional content (kilocalories) and identify categorical food attributes (dish name, ingredients, cooking method) from images. The protocol isolates the value of structured, human-verified data by comparing base foundation models against their supervised fine-tuned counterparts on a frozen test split. Use when the user wants to benchmark on MM-Food-100K, or as...

Evaluates multimodal graph learning models on node classification, link prediction, and knowledge graph completion tasks. It probes the ability of models to integrate high-resolution visual and textual node features with graph structure to make accurate predictions across diverse real-world domains. Use when the user wants to benchmark on Amazon-Sports, Amazon-Cloth, Goodreads-LP, Goodreads-NC, Ele-fashion, MM-CoDEx-s, MM-CoDEx-m, or asks about evaluating this task. Reports MRR, accuracy.

This evaluation probes the effectiveness and efficiency of dynamic sparse attention methods for long-context vision-language models (VLMs). It tests the model's ability to perform long-video understanding, retrieve specific visual or mixed-modality information from extremely long contexts (Needle in a Haystack), and maintain accuracy while reducing computational cost and latency. Use when the user wants to benchmark on Video Understanding Benchmarks, V-NIAH, MM-NIAH, or asks about evaluating ...

Probes human-like abstraction and visual reasoning capabilities in multimodal models across eight fine-grained paradigms, including logical operations, geometry, and spatial relationships. It measures how well models generalize to novel abstract patterns without relying on memorized visual features. Use when the user wants to benchmark on MM-IQ, or asks about evaluating this task. Reports accuracy.

Evaluates the cross-lingual generalization and robustness of Large Vision-Language Models (LVLMs) acting as automated judges. It probes their ability to correctly rank paired multimodal responses across 25 languages while measuring susceptibility to positional and length biases. Use when the user wants to benchmark on MM-JudgeBench, or asks about evaluating this task. Reports average accuracy.

This benchmark evaluates the multimodal diagnostic reasoning capabilities of large vision-language models on brain tumor MRI scans. It probes whether models can integrate subtle visual cues with structured anatomical knowledge to produce accurate diagnoses, while also measuring their ability to recognize uncertainty through explicit rejection options. Use when the user wants to benchmark on MM-NeuroOnco-Bench, or asks about evaluating this task. Reports Accuracy.

Evaluates contextual safety in multi-modal large language models by measuring how well they refuse harmful queries while correctly answering safe ones, with a focus on whether their safety reasoning aligns with the given context. It probes the model's ability to avoid over-defensive refusals on benign inputs while maintaining high response quality. Use when the user wants to benchmark on MM-SafetyBench++, or asks about evaluating this task. Reports Contextual Correctness Rate (CCR).

Evaluates the safety and detoxification capabilities of multimodal large language models by measuring the fraction of harmful responses across various toxicity categories, while also assessing continuous toxicity severity and general multimodal reasoning capability. Use when the user wants to benchmark on MM-SafetyBench, or asks about evaluating this task. Reports Harmful Rate (HaR).

Evaluates vision-language models' ability to perform fine-grained moral reasoning and safety alignment on multimodal scenarios. It probes how well models rank, calibrate, and separate safe from unsafe situations when provided with text, image, or combined modalities. Use when the user wants to benchmark on MM-Scale, or asks about evaluating this task. Reports NDCG@5.

Evaluates Multimodal Large Language Models on low-altitude UAV scenarios, probing their capabilities in visual perception, multi-view spatial reasoning, and egocentric/exocentric planning across diverse real-world aerial imagery tasks. Use when the user wants to benchmark on MM-UAVBench, or asks about evaluating this task. Reports accuracy.

Evaluates Large Multimodal Models' ability to detect unsolvable visual-question problems and appropriately refuse to answer. It specifically probes three failure modes: absent answers, incompatible answer sets, and image-question mismatches. Use when the user wants to benchmark on MM-UPD Bench, or asks about evaluating this task. Reports Dual accuracy.

Evaluates the multi-modal mathematical reasoning capabilities of MLLMs on diverse visual math problems including geometry, charts, and tables. It tests the model's ability to solve multiple-choice and fill-in-the-blank questions using both human-created and synthetically generated unlabeled data. Use when the user wants to benchmark on MathVision, MathVerse, MathVista, We-Math, or asks about evaluating this task. Reports accuracy.

This benchmark evaluates large multimodal models on integrated vision-language capabilities, with a specific focus on sequential image-text understanding, spatial reasoning, knowledge retrieval, and long-form generation. It probes how well models can process interleaved visual and textual inputs to answer complex, real-world questions. Use when the user wants to benchmark on MM-Vet v2, or asks about evaluating this task. Reports MM-Vet-v2 score.

Evaluates the fine-grained localization capability of vision-language models on industrial anomaly detection. It measures how accurately models can predict bounding boxes around defects compared to ground-truth annotations and human experts. Use when the user wants to benchmark on MMAD-BBox, or asks about evaluating this task. Reports BBox-Mask IoU.

Evaluates Multimodal Large Language Models (MLLMs) on industrial anomaly detection tasks, probing their ability to perform fine-grained visual reasoning, multi-image comparison, and defect-related classification, localization, and description. It specifically tests whether models can leverage template normal images and domain knowledge to identify and analyze anomalies in industrial products. Use when the user wants to benchmark on MMAD, or asks about evaluating this task. Reports accuracy.

Evaluates deep audio reasoning capabilities by testing both final answer correctness and the logical quality of intermediate reasoning steps. It covers single-domain (sound, music, speech) and mixed-domain audio tasks to measure how well models avoid spurious correlations and follow verifiable reasoning paths. Use when the user wants to benchmark on MMAR, or asks about evaluating this task. Reports Avg.

Evaluates multilingual passage retrieval models on translated versions of the MS MARCO dataset and the Mr. TyDi dataset. It probes the ability of dense retrieval and reranking models to handle cross-lingual and zero-shot retrieval scenarios, as well as the impact of translation quality on retrieval effectiveness. Use when the user wants to benchmark on mMARCO, Mr. TyDi, or asks about evaluating this task. Reports MRR@10.

Evaluates a generative model's ability to separate target sounds from mixture audio using video and text queries, while preserving video-to-audio generation capabilities. It probes multimodal conditioning, cross-domain knowledge transfer, and the perceptual quality of generated separated audio against discriminative baselines. Use when the user wants to benchmark on VGGSound-Clean, MUSIC, VGGSound, or asks about evaluating this task. Reports FAD.