Claude Skills by qhjqhj00

Evaluates an LLM's ability to extract and structure narrative knowledge from full-length movie screenplays into canonical graphs. It probes entity and relation recognition while filtering out event-centric noise to focus on stable world-building elements. Use when the user wants to benchmark on STAGE-KG, or asks about evaluating this task. Reports Entity F1.

Tests retrieval-augmented question answering over long screenplay contexts. It probes the model's ability to locate relevant information across chunked text and synthesize accurate answers under different retrieval architectures. Use when the user wants to benchmark on STAGE-QA, or asks about evaluating this task. Reports Question Correctness.

Evaluates a model's ability to generate fluent, contextually accurate Japanese image captions directly from visual input. It specifically probes whether native-language training data yields better captioning quality compared to a pipeline of English generation followed by machine translation. Use when the user wants to benchmark on STAIR Captions, or asks about evaluating this task. Reports CIDEr.

This benchmark evaluates a model's ability to classify the stance (InFavor, Against, or None) of text towards a specific target or query. It specifically probes cross-target generalization by training on multiple targets and testing on a held-out target, while also assessing robustness to sarcastic or figurative language through intermediate sarcasm pre-training. Use when the user wants to benchmark on SemEval 2016 Task 6A Dataset, Multi-Perspective Consumer Health Query Data (MPCHI), or asks...

Evaluates a model's ability to classify stance (in favor, against, neutral) toward climate change and related targets, while jointly predicting sentiment (positive, negative, neutral). It probes the synergy between stance classification and sentiment analysis using text and topic features. Use when the user wants to benchmark on Climate Change Tweets Dataset, SemEval-2016 Task 6.A, or asks about evaluating this task. Reports F1 score.

Evaluates language model capabilities across knowledge, reasoning, instruction following, and safety using a standard suite of downstream benchmarks. It measures both pretraining quality and post-adaptation performance on established NLP and coding tasks. Use when the user wants to benchmark on MMLU, HellaSwag, ARC-Challenge, ARC-Easy, PIQA, WinoGrande, GSM8k, BBH, HumanEval, AlpacaEval 1.0, XSTest, IFEval, or asks about evaluating this task. Reports exact-match accuracy.

Evaluates 2D-to-3D semantic transfer pipelines for indoor scene understanding, focusing on per-point labeling accuracy for structural and furniture classes, and detection sensitivity for novel safety-critical objects in public safety contexts. Use when the user wants to benchmark on Stanford 2D-3D-S*, or asks about evaluating this task. Reports per-point accuracy.

Evaluates the ability of models to recover 3D geometry and surface material from images, and to synthesize novel views or relight objects in unseen real-world environments. It probes inverse rendering capabilities under natural, uncontrolled lighting conditions where ground-truth material is unavailable. Use when the user wants to benchmark on Stanford-ORB, or asks about evaluating this task. Reports Bidirectional Chamfer Distance.

Evaluates a model's ability to perform spatio-temporal reasoning and answer questions about dynamic scenes using compressed textual scene graph sequences. It probes the model's capacity to track object interactions, understand event ordering, and generalize to unseen temporal compositions without relying on raw visual inputs. Use when the user wants to benchmark on STAR, AGQA, or asks about evaluating this task. Reports Accuracy.

Evaluates an LLM's ability to generate correct SQL queries from natural language questions over complex, multi-table database schemas. It probes the model's reasoning capabilities and schema generalization by requiring step-by-step rationales and testing on unseen databases. Use when the user wants to benchmark on Spider, or asks about evaluating this task. Reports execution accuracy (EX).

Evaluates code generation, completion, and bug-fixing capabilities across multiple programming languages and libraries. It probes a model's ability to write correct functions from prompts, translate code across languages, and fix existing buggy code using standard and enhanced benchmarks. Use when the user wants to benchmark on HumanEval, MBPP, EvalPlus, MultiPL-E, DS-1000, HumanEvalFix, or asks about evaluating this task. Reports pass@1.

Evaluates vision-language models' ability to parse free-form workflow sketch images and generate structured JSON workflow definitions. It probes structural fidelity, trigger and component recognition, and hierarchical consistency in diagram-to-code translation. Use when the user wants to benchmark on StarFlow Dataset, or asks about evaluating this task. Reports FlowSim.

Evaluates retrieval models on their ability to find relevant entities in semi-structured knowledge bases using complex queries that combine textual descriptions and relational constraints. It probes joint reasoning over mixed textual-relational semantics and user-intent modeling across product, academic, and medical domains. Use when the user wants to benchmark on STaRK, or asks about evaluating this task. Reports Hit@k.

Evaluates whether a single Vision-Language-Action model can generalize across diverse robotic manipulation benchmarks without task-specific fine-tuning. It probes the model's cross-embodiment generalization and robustness to varying action spaces and task distributions. Use when the user wants to benchmark on LIBERO, SimplerEnv, RoboTwin 2.0, RoboCasa-GR1, RoboChallenge, or asks about evaluating this task. Reports success_rate.

Evaluates a model's ability to retrieve relevant statistical data tables from a large corpus based on conversational dialogue history, and its ability to generate appropriate agent responses. It probes intent understanding, table-level grounding, and robustness to temporal distribution shifts. Use when the user wants to benchmark on StatCan Dialogue Dataset, or asks about evaluating this task. Reports recall@10.

Evaluates multimodal agents' ability to perceive current GUI states from screenshots, interpret natural language toggle instructions, and execute precise click actions. It specifically probes state-aware reasoning by measuring accuracy on both positive and negative toggle instructions, as well as grounding precision and false positive/negative rates. Use when the user wants to benchmark on state control benchmark, dynamic evaluation benchmark, or asks about evaluating this task. Reports O-AMR.

Evaluates whether LLMs can track dynamic states over sequential update instructions. It probes the model's ability to maintain and update internal representations of an environment's state across multiple steps, testing sequential reasoning and input-window memory limits. Use when the user wants to benchmark on State-Tracking-Tasks (LinearWorld, HandSwap, Lights), or asks about evaluating this task. Reports accuracy.

Compute the StatScores metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute StatScores, or asks how to score with StatScores.

This benchmark evaluates whether deep learning models can accurately predict the epicentral distance of an earthquake from single-station ground motion waveforms. It specifically probes whether models learn intrinsic seismic features or merely exploit highly correlated auxiliary signals like P/S wave arrival times. Use when the user wants to benchmark on Stanford Earthquake Dataset (STEAD), or asks about evaluating this task. Reports Mean Absolute Error (MAE).

Evaluates a deep learning model's ability to directly estimate earthquake magnitude (both local ML and duration Md) from raw, unprocessed single-station seismograms without normalization or instrument response correction. It probes the model's robustness to site effects, regional calibration differences, and varying signal-to-noise ratios. Use when the user wants to benchmark on STEAD, or asks about evaluating this task. Reports mean error.

This evaluation probes a decision-maker's ability to stealthily sample a subset of a dataset to artificially satisfy fairness metrics (Demographic Parity) while remaining statistically indistinguishable from the original data distribution. It measures how well biased sampling algorithms can evade detection by ideal auditors using distributional tests like Kolmogorov-Smirnov or Wasserstein distance. Use when the user wants to benchmark on Synthetic Loan Check, COMPAS, Adult, or asks about eval...

Evaluates the safety and helpfulness alignment of multimodal large language models under single-turn versus multi-turn interactive settings, specifically probing the static-to-dynamic generalization gap and the evolution of safety failure rates across conversation turns. Use when the user wants to benchmark on Steer-Bench, or asks about evaluating this task. Reports pass_rate.

This framework evaluates the effectiveness of representation steering methods in modifying specific safety behaviors (harmfulness, hallucination, bias) while measuring cross-perspective entanglement. It probes whether steering interventions achieve their target behavioral changes without causing unintended degradation in other safety or reasoning capabilities. Use when the user wants to benchmark on SteeringSafety Benchmark (17 datasets), or asks about evaluating this task. Reports effectiven...

Evaluates academic search and recommendation systems in live production environments using A/B testing and user interaction logs, bridging the gap between offline test collections and real-world performance. Use when the user wants to benchmark on LIVIVO, GESIS Search, or asks about evaluating this task. Reports click-paths.

Evaluates sequential recommendation models that incorporate social influence and temporal dynamics. It probes the ability to predict the next item a user will interact with based on their historical behavior sequence, social connections, and event timestamps. Use when the user wants to benchmark on Delicious, Yelp, Ciao, or asks about evaluating this task. Reports Recall@10.

Probes a model's capability to accurately edit or synthesize audio with specific emotional tones, speaking styles, and paralinguistic elements (e.g., laughter, sighs) using zero-shot voice cloning or iterative refinement. It measures how well the model preserves linguistic content while transferring or inserting target audio attributes. Use when the user wants to benchmark on Step-Audio-Edit-Test, or asks about evaluating this task. Reports accuracy.

Evaluates audio language models on speech understanding, reasoning, and real-time interactive dialogue capabilities using raw acoustic signals rather than textual transcriptions. It measures both comprehension accuracy across multiple audio benchmarks and real-time generation fluency. Use when the user wants to benchmark on Big Bench Audio, Spoken MQA, MMSU, MMAU, Wild Speech, or asks about evaluating this task. Reports Average Score (%).

Evaluates a vision-language model's ability to perceive, locate, and interact with graphical user interfaces across desktop and mobile environments. It also measures general multimodal reasoning and OCR capabilities to ensure the model retains broad foundational skills after GUI-specific training. Use when the user wants to benchmark on ScreenSpot-Pro, ScreenSpot-v2, OSWorld-G, MMBench-GUI-L2, VisualWebBench, OSWorld-Verified, AndroidWorld, AndroidDaily, or asks about evaluating this task. Re...

Evaluates the ability of a stereo video deblurring algorithm to recover sharp frames from motion-blurred inputs, specifically testing its robustness to spatially-variant blur caused by independent 3D object motion and non-planar surfaces. Use when the user wants to benchmark on Custom synthetic raytraced & real stereo captures, or asks about evaluating this task. Reports PSNR.

Evaluates 6D object pose estimation methods under stereo vision conditions, specifically probing robustness to occlusion and scale ambiguity by leveraging dense 2D-3D correspondences and stereo disparity. Use when the user wants to benchmark on Stereo PBR YCB-V DS, or asks about evaluating this task. Reports ADD0.1.

Evaluates stereotypical bias in pretrained language models across gender, profession, race, and religion using context-based association tests. It measures both language modeling capability and the model's tendency to prefer stereotypical over anti-stereotypical associations in natural language contexts. Use when the user wants to benchmark on StereoSet, or asks about evaluating this task. Reports icat.

Evaluates a model's ability to recover coherent source time functions (STFs) from scattered, noisy seismic wavefields without relying on traditional deconvolution or labeled seismograms. Use when the user wants to benchmark on Synthetic Scattering Simulation, or asks about evaluating this task. Reports maximum normalized cross-correlation (MNCC).

Evaluates the multi-step spatio-temporal forecasting capability of graph neural networks on urban traffic speed and flow prediction tasks. It measures how well models capture spatial dependencies and temporal dynamics across varying prediction horizons (3, 6, and 12 steps). Use when the user wants to benchmark on METR-LA, PEMS-BAY, PEMSD4, PEMSD8, or asks about evaluating this task. Reports MAE.

This evaluation protocol tests a model's ability to forecast future traffic flow across large-scale urban road networks using historical spatiotemporal sensor data. It probes the model's capacity to capture complex spatial dependencies and temporal dynamics while maintaining computational efficiency on real-world traffic benchmarks. Use when the user wants to benchmark on LargeST, PEMS-series, or asks about evaluating this task. Reports MAE.

Evaluates the ability of uncertainty-aware deep neural networks to approximate a highly irregular, multi-extremum function and quantify predictive uncertainty. It specifically probes how well the models handle in-distribution versus out-of-distribution parameter regimes. Use when the user wants to benchmark on Stochastic Ackley Function, or asks about evaluating this task. Reports Relative Error (RE).

Evaluates the ability of a generative diffusion model to emulate kilometer-scale atmospheric convection and precipitation forecasting. It probes spatial-temporal forecast skill, multi-variable physical consistency (e.g., updrafts, cold pools), and spectral fidelity over 1-6 hour lead times. Use when the user wants to benchmark on ERA5, HRRR, MRMS, or asks about evaluating this task. Reports FSS.

Evaluates a spatio-temporal graph neural network's ability to predict and correct systematic biases in storm surge water level forecasts. It probes the model's capacity to leverage spatial dependencies among coastal gauge stations and temporal patterns to improve long-horizon (up to 72h) hydrodynamic predictions. Use when the user wants to benchmark on Gulf Coast Gauge Network (NOAA/TCOON), or asks about evaluating this task. Reports RMSE.

Evaluates multimodal capabilities across visual understanding, speech interaction, and vision-grounded speech tasks. It measures accuracy on standard VQA and knowledge-grounded QA benchmarks, and uses LLM-based scoring for open-ended spoken interactions. Use when the user wants to benchmark on VQA-v2, GQA, VizWiz, ScienceQA-IMG, TextVQA, POPE, MME, MMBench, SEED-Bench, LLaVA-Bench-in-the-Wild, MM-Vet, Llama Questions, Web Questions, SpokenVisIT, or asks about evaluating this task. Reports acc...

Evaluates the spatial realism and temporal flow consistency of videos generated by video generative models. It decomposes video features into spatial and temporal components using embedding spaces and Fourier transforms to provide length-agnostic, bounded scores that independently measure visual fidelity and motion naturalness. Use when the user has predictions and gold and needs to compute STREAM-T.

Evaluates real-time streaming video understanding and multi-turn dialogue capabilities. It measures semantic correctness, factual accuracy, dialogue coherence, and system latency across diverse video types and query formats. Use when the user wants to benchmark on STREAMBENCH, or asks about evaluating this task. Reports Acc..

Evaluates hydrological forecasting models on predicting streamflow across 319 US basins. It tests the model's ability to capture long-term temporal dependencies and intermediate hydrological states (soil water, snowpack) under varying data segmentation, training sizes, and noise conditions. Use when the user wants to benchmark on 319 US Basins, or asks about evaluating this task. Reports RMSE.

Evaluates a model's ability to perform streaming camera pose estimation and 3D reconstruction over long video sequences. It probes long-range geometric consistency, drift resistance, and reconstruction fidelity across diverse indoor and outdoor environments. Use when the user wants to benchmark on Oxford Spires, ETH3D, 7-Scenes, Tanks and Temples, NRGBD, or asks about evaluating this task. Reports ATE, F1.

Evaluates multimodal large language models' ability to understand real-time streaming video, integrate visual and audio information, maintain contextual continuity across sequential questions, and proactively output information at specific timestamps. Use when the user wants to benchmark on StreamingBench, or asks about evaluating this task. Reports accuracy.

This benchmark evaluates the imperceptibility, robustness to benign audio transformations, and semi-fragility to malicious deepfake manipulations of a deep learning-based audio watermarking system. It specifically probes whether a watermark can survive standard compression and cropping while being deliberately destroyed by semantic-altering AI conversions like voice cloning or speech editing. Use when the user wants to benchmark on LibriSpeech (train_clean100), Test Set A, Test Set B, or asks...

Evaluates the ability of long-term agent memory compression methods to retain and retrieve critical information from streaming and long-form videos. It probes how effectively a model can maintain temporal consistency and answer complex queries while significantly reducing memory graph size, measured via answer accuracy across robotic, web, and general video benchmarks. Use when the user wants to benchmark on M3-Bench-robot, M3-Bench-web, Video-MME-Long, or asks about evaluating this task. Rep...

Evaluates real-time speech translation systems on latency and translation quality across multiple language pairs. It probes the model's ability to dynamically decide when to generate and truncate translations while processing streaming audio inputs. Use when the user wants to benchmark on MuST-C English→German, MuST-C English→Spanish, CoVoST2 English→Chinese, CoVoST2 French→English, or asks about evaluating this task. Reports SacreBLEU, COMET.

Tests an LLM agent's real-time decision-making and combat strategy in a video game environment, prioritizing low latency while maintaining competitive win rates. The benchmark probes the model's ability to make timely character actions under a hard frame-rate limit. Use when the user wants to benchmark on StreetFighter, or asks about evaluating this task. Reports ELO Score.

This benchmark evaluates the quality of differentially private synthetic structured text generation by measuring how well synthetic datasets preserve the syntactic structure, semantic dependencies, and attribute distributions of real data, alongside downstream task utility. Use when the user wants to benchmark on ShareGPT, or asks about evaluating this task. Reports CFG Pass Rate (CFG-PR).

Compute the StructuralSimilarityIndexMeasure metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute StructuralSimilarityIndexMeasure, or asks how to score with StructuralSimilarityIndexMeasure.

Evaluates the ability of LLMs to generate executable structured queries (SQL, SPARQL) from natural language questions over tables, knowledge graphs, and temporal knowledge graphs. It specifically probes the model's capacity for iterative self-correction when initial query generation fails, measuring both raw generation accuracy and error-resilience during multi-round inference. Use when the user wants to benchmark on WikiSQL, WTQ, MetaQA, WebQSP, CronQuestions, TabFact, or asks about evaluati...