Claude Skills by qhjqhj00

Evaluates large language models' ability to perform mathematical, commonsense, and natural language inference reasoning in low-, medium-, and high-resource languages. It specifically probes cross-lingual transfer capabilities using a zero-shot chain-of-thought setting without requiring parallel multilingual instruction data. Use when the user wants to benchmark on MMWP, MGSM, MSVAMP, X-CSQA, XNLI, or asks about evaluating this task. Reports accuracy.

Evaluates multi-modal vision-language models on their ability to perform visual question answering across two temporal X-ray images to detect regional disease progression. It probes temporal reasoning, subtle change detection, and bias mitigation in medical imaging diagnostics. Use when the user wants to benchmark on MMXU-test, or asks about evaluating this task. Reports accuracy.

Evaluates a model's ability to learn sequentially from multiple tasks without catastrophic forgetting across three distinct continual learning splits of MNIST: incremental task, domain, and class learning. It measures how well baseline and continual learning methods retain knowledge from previous tasks while adapting to new ones. Use when the user wants to benchmark on MNIST, or asks about evaluating this task. Reports average accuracy (%).

Evaluates the out-of-domain generalization and robustness of NLI models trained under different data collection protocols. It measures how well models perform on held-out, genre-diverse, and adversarial benchmarks compared to in-domain validation performance. Use when the user wants to benchmark on MNLI-mismatched, ANLI, or asks about evaluating this task. Reports accuracy.

Evaluates multilingual neural machine translation performance across many-to-one, one-to-many, and many-to-many translation scenarios, testing how dynamic parameter differentiation impacts translation quality across diverse language pairs and resource levels. Use when the user wants to benchmark on OPUS, WMT, IWSLT'17, or asks about evaluating this task. Reports BLEU.

Evaluates the inference latency and computational efficiency of mobile deep learning engines across different hardware platforms, compute backends, and neural network architectures. Use when the user wants to benchmark on MobileNet-v1, SqueezeNet-v1.1, ResNet-18, Inception-v3, or asks about evaluating this task. Reports inference time (ms).

Evaluates the ability of speech recognition models to jointly transcribe Mandarin speech and identify nonverbal vocalizations (NVs) like laughs or sighs. It also isolates the strict accuracy of NV event detection and measures whether adding NV recognition degrades core lexical transcription performance. Use when the user wants to benchmark on MNV-17, or asks about evaluating this task. Reports CER.

Evaluates the ability of automated theorem provers and LLMs to formally prove complex algebraic inequalities at the International Mathematical Olympiad level using a deductive search engine in Lean. Use when the user wants to benchmark on MO-INT-20, or asks about evaluating this task. Reports number of solved problems.

Evaluates geospatial representation models on 11 regression tasks spanning social, economic, and natural domains across multiple spatial scales (point, grid, county, city). It probes the model's ability to capture complex human-centric and environmental patterns using fused multimodal embeddings rather than relying solely on geographic coordinates. Use when the user wants to benchmark on MobCLIP Downstream Benchmark (11 tasks), or asks about evaluating this task. Reports R^2.

Evaluates mobile GUI agents' ability to complete tasks on mobile interfaces by measuring task success rate across diverse, multi-path trajectories. It also enables modular component analysis (screen parsing, history generation, inference style, reflection) to identify performance bottlenecks and optimal configurations for different foundation models. Use when the user wants to benchmark on MobiBench, or asks about evaluating this task. Reports Task Success Rate (TSR).

This benchmark evaluates the capability of visual tracking algorithms to maintain robust face localization in unconstrained, mobile-captured video sequences. It specifically probes resilience to challenging real-world conditions such as rapid camera motion, out-of-plane rotations, scale changes, and partial occlusions. Use when the user wants to benchmark on iBUG MobiFace, or asks about evaluating this task. Reports AUC.

Evaluates the accuracy and robustness of lightweight face recognition models on unconstrained face verification tasks, specifically measuring pair-wise verification accuracy and identification rates under extreme distractor conditions. Use when the user wants to benchmark on Labeled Faces in the Wild (LFW), MegaFace, or asks about evaluating this task. Reports Accuracy, TAR@FAR=10^-6.

Probes an agent's ability to navigate and execute multi-step UI operations on real mobile devices (Android/HarmonyOS) based on natural language instructions. It evaluates end-to-end task completion, step-level correctness, decision-making precision, and the capacity to detect and correct operational errors via reflection. Use when the user wants to benchmark on Mobile-Agent-v2 Evaluation Set, or asks about evaluating this task. Reports Success Rate (SR).

Evaluates LLM-based mobile agents on real-world task execution across single-app and multi-app scenarios. It probes the agent's ability to plan, navigate UIs, call APIs, and collaborate across applications to complete user-defined goals. Use when the user wants to benchmark on Mobile-Bench, or asks about evaluating this task. Reports PassRate.

Tests a robot's ability to simultaneously navigate and construct a target structure in 1D/2D/3D grid worlds under partial observability and environmental uncertainty. It evaluates the bi-directional coupling between localization and manipulation planning. Use when the user wants to benchmark on Mobile Construction Benchmark, or asks about evaluating this task. Reports IoU.

Evaluates the effectiveness of parallel deep learning inference strategies across heterogeneous mobile processors (CPU, GPU, DSP) under varying workloads and dynamic system conditions. It probes how operator support, scheduling granularity, and competing processes impact inference latency, resource utilization, and system responsiveness. Use when the user wants to benchmark on Standard DL Models (YOLOv2, VGG-16, PoseNet, FST, RetinaFace, ResNet-18, ResNet-50), or asks about evaluating this ta...

This evaluation probes the hardware efficiency and resource constraints of training deep learning models on mobile SoCs. It measures how different model architectures and batch sizes impact GPU/CPU utilization, power/energy draw, and memory footprint during training. Use when the user has predictions and gold and needs to compute GPU utilization.

Evaluates an autonomous mobile device agent's ability to execute multi-step UI operations using visual perception. It probes task planning, self-reflection, and cross-application interaction under varying instruction complexity. Use when the user wants to benchmark on Mobile-Eval, or asks about evaluating this task. Reports Success (Su).

This evaluation probes a model's ability to recognize hand gestures from mobile device inertial sensor data (accelerometer and gyroscope). It measures classification robustness across varying signal speeds, amplitudes, and noise levels by testing on three distinct datasets with different gesture sets and sensor configurations. Use when the user wants to benchmark on MGD, BUAA Mobile Gesture Database, SmartWatch Gesture Database, or asks about evaluating this task. Reports classification accur...

This benchmark evaluates the inference performance of a mobile GPU-accelerated neural network engine. It measures execution latency and peak memory consumption across different hardware platforms, precision formats (FP32 vs FP16), and batch sizes. It probes the engine's ability to leverage GPU parallelism and half-precision arithmetic for efficient edge deployment. Use when the user wants to benchmark on MNIST, Cifar-10, Style Transfer Dataset, or asks about evaluating this task. Reports exec...

Evaluates the capability of multimodal large language models to act as autonomous mobile GUI agents. It probes their ability to plan tasks, predict correct interaction types, accurately ground UI elements, and successfully complete complex, multi-step workflows in both static and dynamic Android environments. Use when the user wants to benchmark on AndroidControl, AndroidLab, Android Agent Arena (A3), or asks about evaluating this task. Reports Success Rate (SR).

Evaluates the practical feasibility of running deep learning models on mobile hardware by measuring end-to-end latency and energy consumption during CNN inference. It compares on-device execution against cloud-based execution to identify hardware and network bottlenecks. Use when the user wants to benchmark on Mobile Benchmark Image Set, or asks about evaluating this task. Reports end-to-end latency.

Evaluates language models' understanding of mobile-specific domains and tasks under on-device constraints. It probes the models' ability to answer multiple-choice questions across 80 real-world mobile domains, emphasizing practical usability, privacy, and personalization in daily mobile interactions. Use when the user wants to benchmark on Mobile-MMLU, or asks about evaluating this task. Reports accuracy.

Evaluates a compact on-device unified vision-language-diffusion model's capabilities in multimodal understanding, text-to-image generation, and image editing. It probes the model's ability to align cross-modal representations and generate high-fidelity images while maintaining real-time inference speeds on edge hardware. Use when the user wants to benchmark on GenEval, MMMU, MM-Vet, SEED, TextVQA, ChartQA, POPE, GQA, ImageEdit, or asks about evaluating this task. Reports GenEval overall score.

Evaluates a vision-language model's ability to navigate and complete multi-step tasks on mobile GUIs. It measures step-level action correctness, full trajectory success, and robustness to intermediate errors in a simulated Android environment. Use when the user wants to benchmark on Chinese Mobile Agent Benchmark, or asks about evaluating this task. Reports Accuracy (Acc.).

Evaluates a CNN model's ability to classify smartphone-captured images of skin lesions into one of seven dermatological conditions. It probes the model's robustness to class imbalance and the effectiveness of data preprocessing strategies like oversampling and augmentation. Use when the user wants to benchmark on Smartphone Skin Disease Dataset, or asks about evaluating this task. Reports accuracy.

Evaluates the ability of deep learning models to forecast multi-service mobile network traffic volumes at the antenna level over short time horizons (up to 1 hour). It probes spatiotemporal sequence modeling by requiring models to capture both spatial correlations across antennas and temporal dependencies in traffic patterns. Use when the user wants to benchmark on Real-world mobile traffic dataset (36 services, 800 antennas), or asks about evaluating this task. Reports MAE.

Evaluates the runtime efficiency, hardware utilization, and thermal/energy impact of deploying vision-language models on mobile devices. It measures latency breakdowns, CPU/GPU/NPU usage, power consumption, and output characteristics across different inference frameworks. Use when the user wants to benchmark on Custom Mobile VLM Inference Test Set, or asks about evaluating this task. Reports Latency.

This benchmark evaluates the performance of LLM-based mobile agents on Android GUI navigation tasks. It measures end-to-end task completion, action efficiency, response latency, computational cost, and the agent's ability to correctly determine task completion without stopping too early or too late. Use when the user wants to benchmark on MobileAgentBench, or asks about evaluating this task. Reports Success Rate (SR).

Evaluates the task performance, latency, and hardware utilization of quantized LLMs and LMMs on real mobile devices across standard NLP, multi-modal, and trust & safety benchmarks. Use when the user wants to benchmark on Databricks, HotpotQA, sql-create-context, CNN, XSum, VQA-v2, GQA, VisWiz, TextVQA, SQA, AlpacaEval, MT-Bench, MMLU, GSM8K, TruthQA, BBQ, SC-101, Adv-Inst, DNA, Priv-Lk, or asks about evaluating this task. Reports Accuracy.

Evaluates a lightweight hybrid deep learning model for classifying chest X-ray images into COVID-19, Normal, and Pneumonia categories, and optionally predicting disease severity scores. Use when the user wants to benchmark on COVIDx, or asks about evaluating this task. Reports F1 Score.

Evaluates the visual quality and generation speed of image-to-video diffusion models optimized for mobile deployment. It probes the model's ability to generate temporally coherent 17-frame videos from a single reference image while maintaining high resolution and low latency on mobile hardware. Use when the user wants to benchmark on Unspecified (FVD benchmarks), or asks about evaluating this task. Reports FVDhum.

Evaluates LLMs' capability to generate syntactically valid, functionally correct, and hardware-efficient C/C++ kernels for mobile inference engines. It probes framework-specific API usage, compilation robustness, functional verification against ONNX baselines, and runtime speedup optimization. Use when the user wants to benchmark on MobileKernelBench, or asks about evaluating this task. Reports Compilation success rate (CSR).

Evaluates autonomous mobile agents on long-horizon, cross-application workflows, requiring them to handle ambiguous instructions via agent-user interaction and integrate external tools via MCP. It probes planning, GUI grounding, clarification strategies, and tool orchestration in real-world mobile environments. Use when the user wants to benchmark on MobileWorld, or asks about evaluating this task. Reports Success Rate (SR).

This benchmark evaluates the accuracy of deep learning models for multivariate time series forecasting of aggregated human mobility across six urban regions. It specifically probes how well models can predict short-term (30-minute) passenger counts while preserving differential privacy through input or gradient perturbation mechanisms. Use when the user wants to benchmark on Paris Mobility Dataset, or asks about evaluating this task. Reports RMSE.

Evaluates the accuracy of time series forecasting models on urban mobility data across different prediction horizons. It probes how well traditional, deep learning, and foundation models capture short-term, medium-term, and long-term temporal dependencies in bike-sharing flows. Use when the user wants to benchmark on BikeNYC, BikeVIE, or asks about evaluating this task. Reports RMSE.

Evaluates LLM-based route-planning agents on real-world mobility queries, probing their ability to handle multi-waypoint itineraries, preference-constrained routing, and multimodal travel. It measures how well agents understand instructions, decompose tasks, select tools, and produce valid, constraint-satisfying routes. Use when the user wants to benchmark on MobilityBench, or asks about evaluating this task. Reports Final Pass Rate (FPR).

Evaluates whether video multi-modal LLMs genuinely utilize motion cues for pixel-level visual grounding. It specifically probes their ability to distinguish true motion from static fake motion (Motion Existence) and to differentiate forward from reversed motion sequences (Motion Order). Use when the user wants to benchmark on MoCentric-Bench, or asks about evaluating this task. Reports mIoU.

Evaluates a model's ability to mitigate spurious correlations (bias) in image classification by measuring performance on both overall test sets and specifically on bias-conflicting samples where the spurious attribute contradicts the true label. Use when the user wants to benchmark on Corrupted CIFAR-10, BAR, BFFHQ, Waterbirds, or asks about evaluating this task. Reports Average Accuracy.

This evaluation probes a model's ability to automatically detect and classify sample-specific modality bias in multimodal misinformation content. It measures how well automated quantification methods align with human judgment regarding whether a sample relies on image-only, text-only, or balanced modalities. Use when the user wants to benchmark on Fakeddit, MMFakeBench, or asks about evaluating this task. Reports Accuracy.

Evaluates a graph neural network's ability to detect structural anomalies by reconstructing multivariate time-series sensor data. It measures how well the model captures physical material properties and eigenmode shifts compared to spectral GNN baselines, while also benchmarking computational efficiency. Use when the user wants to benchmark on Luxembourg dataset, Simulated Smart Bridge dataset, or asks about evaluating this task. Reports reconstruction error.

This benchmark evaluates the accuracy and hardware efficiency of neural flow-based architectures for recovering underlying dynamics from time-series data. It probes the model's ability to estimate parameters of nonlinear dynamical systems while measuring computational resource constraints like runtime, power, and memory footprint on edge hardware. Use when the user wants to benchmark on Chaotic Lorenz, F8 Cruiser, Lotka Volterra, Pathogenic Attack System, Automated Insulin Delivery (OhioT1D),...

This evaluation protocol assesses the trade-off between model size (parameter count) and task utility across multiple AI benchmarks. It aims to identify energy-efficient models that maintain high performance, enabling estimation of global AI inference energy savings through strategic model selection. Use when the user wants to benchmark on OpenLLM Leaderboard, LMSys Chatbot Arena, NPHardEval, BigCode Leaderboard, mtebLeaderboard, WMT English-German, Open Object Detection Leaderboard, ImageNet...

Evaluates the inference latency and end-to-end turn-around time of five machine learning model-serving frameworks across four distinct real-world inference scenarios. It probes how framework specialization (DL-specific vs. general-purpose) and input payload size affect serving performance and stability. Use when the user wants to benchmark on Malware detection, Cryptocoin price forecasting, Image classification, Sentiment analysis, or asks about evaluating this task. Reports average_latency.

Tests whether language models exhibit specific emergent or undesirable behaviors (e.g., sycophancy, self-preservation, political bias) by measuring their preference for behavior-matching versus behavior-mismatching labels. It evaluates how well smaller or preference-trained models predict the behavioral tendencies of larger or RLHF-trained counterparts. Use when the user wants to benchmark on Model-Written Evaluations (133 behaviors), or asks about evaluating this task. Reports accuracy.

Evaluates the ability of retrieval systems to find relevant AI model tables from a heterogeneous corpus. It probes semantic understanding of structured data and cross-source table matching. Use when the user wants to benchmark on ModelTables, or asks about evaluating this task. Reports P@1.

Evaluates the precision and accuracy of stellar flux recovery and positional measurements in simulated Roman Space Telescope images. It probes how well effective PSF models can recover input fluxes and coordinates across different spatial grids, filters, and magnitudes. Use when the user has predictions and gold and needs to compute modified median absolute deviation ($\hat{\sigma}$).

Compute the ModifiedPanopticQuality metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute ModifiedPanopticQuality, or asks how to score with ModifiedPanopticQuality.

Evaluates the predictive accuracy and uncertainty quantification of a descriptor-based neural network on materials property prediction tasks, with a focus on low-data regimes and bias-imbalance. Use when the user wants to benchmark on Naccarato et al. refractive index, Petretto et al. vibrational thermodynamics, or asks about evaluating this task. Reports mean absolute error.

Measures stereotypical bias in large vision-language models across gender, race, and occupational/persona attributes. It evaluates how model outputs deviate from real-world demographic baselines or equal distribution when presented with visual inputs paired with text prompts. Use when the user wants to benchmark on UTKFace, SD-v2.1 Generated (Persona Traits), or asks about evaluating this task. Reports stereotypical_bias.