Claude Skills by qhjqhj00

Evaluates Automatic Post-Editing (APE) systems by measuring how well they correct machine-translated German sentences using monolingual and bilingual neural translation models combined via log-linear weighting. Use when the user wants to benchmark on WMT 2016 APE Shared Task Development Set, or asks about evaluating this task. Reports TER.

Evaluates neural machine translation systems across multiple language pairs (EN-DE, EN-CS, CS-EN, EN-RO, RO-EN, EN-RU, RU-EN) on news text. It measures translation quality using BLEU scores on held-out test sets to assess the impact of techniques like back-translation, ensembling, and subword segmentation. Use when the user wants to benchmark on WMT 2016 News Translation, or asks about evaluating this task. Reports BLEU.

Evaluates machine translation systems on discourse-level literary translation from Chinese to English, focusing on long-range context, cultural adaptation, and stylistic fidelity across entire web novels rather than isolated sentences. Use when the user wants to benchmark on WMT 2023 Discourse-Level Literary Translation Test Set, or asks about evaluating this task. Reports d-BLEU.

Evaluates machine translation quality estimation across sentence-level scoring, word-level error detection, and fine-grained error span classification. It probes a model's ability to predict translation quality and localize specific errors without relying on human references during inference. Use when the user wants to benchmark on WMT2022 QE EN-DE dataset, WMT2022 Metric EN-DE dataset, WMT17/19/20 Post-editing EN-DE datasets, or asks about evaluating this task. Reports MCC.

Evaluates document-level and literary-domain machine translation quality, focusing on discourse-aware metrics like consistency, anaphora resolution, and term consistency across long-form Chinese web novels. The protocol combines automated n-gram and neural metrics with a structured human judgment framework to capture cross-sentence coherence and literary style. Use when the user wants to benchmark on WMT 2024 Discourse-Level Literary Translation Shared Task, or asks about evaluating this task...

Evaluates Chinese-to-English machine translation performance on biomedical texts. It measures how well a neural MT system can translate domain-specific terminology and syntax while handling case, punctuation, and subword tokenization conventions. Use when the user wants to benchmark on WMT21 OK-aligned biomedical test set, or asks about evaluating this task. Reports BLEU.

Evaluates the translation quality and inference speed of non-autoregressive versus autoregressive machine translation models on English-German news text. It probes the practical trade-offs between decoding latency and translation accuracy under realistic deployment conditions. Use when the user wants to benchmark on WMT21 News Translation, or asks about evaluating this task. Reports BLEU.

Evaluates neural machine translation performance across news and biomedical domains for English-German and English-Russian language pairs. It probes the model's ability to handle domain-specific vocabulary, cross-lingual alignment, and translation quality under constrained data conditions typical of shared task tracks. Use when the user wants to benchmark on WMT21 News & Biomedical Shared Tasks, or asks about evaluating this task. Reports BLEU.

Evaluates machine translation quality estimation systems by predicting human judgments on translation adequacy and fluency (Direct Assessment) and classifying translation errors (CED). It probes the model's ability to correlate predicted scores with human ratings and accurately detect translation quality issues across multiple language pairs. Use when the user wants to benchmark on WMT 2021 Quality Estimation Shared Task datasets, or asks about evaluating this task. Reports Pearson's correlat...

Evaluates sign language translation from video to spoken text. It probes the model's ability to handle long videos, large vocabularies, and high singleton rates by leveraging full-body and lip-reading visual features. Use when the user wants to benchmark on WMT 2022 Shared Task, PHOENIX 2014T, or asks about evaluating this task. Reports BLEU.

Evaluates machine translation models on English-to-multiple-target-language pairs using standard development sets. It measures translation quality via BLEU scores to compare bilingual versus multilingual decoder representations and capacity. Use when the user wants to benchmark on WMT22 General Machine Translation, Multitarget TED talks, or asks about evaluating this task. Reports BLEU.

Evaluates machine translation systems across 55 languages and dialects using automatic metrics and significance testing to compare translation quality across different domains and language pairs. Use when the user wants to benchmark on WMT24++, or asks about evaluating this task. Reports BLEU.

Evaluates machine translation systems for bilingual customer support conversations, focusing on context utilization, discourse coherence, and turn-level versus conversation-level translation quality across five language pairs. Use when the user wants to benchmark on MAIA 2.0, or asks about evaluating this task. Reports COMET.

Evaluates document-level machine translation quality using multi-turn conversational prompting strategies with LLMs, measuring contextual coherence and translation accuracy across multiple language directions and domains. Use when the user wants to benchmark on WMT 24 General Track, WMT 23 Chinese-to-English, or asks about evaluating this task. Reports dBLEU.

Evaluates machine translation quality for low-resource Northeast Indian languages (Assamese, Khasi, Mizo, Manipuri) paired with English. It probes cross-lingual transfer capabilities, model adaptation under data scarcity, and the effectiveness of architectural constraints like layer freezing and script-based language grouping. Use when the user wants to benchmark on IndicNECorp1.0, or asks about evaluating this task. Reports BLEU.

Evaluates the ability of LLMs to accurately assess machine translation quality across varying input lengths (segment, document, and long-form). It probes whether LLMs can maintain consistent error detection and system ranking accuracy when processing longer texts, and tests prompting/fine-tuning strategies to mitigate length bias. Use when the user wants to benchmark on WMT'24 metrics shared task, or asks about evaluating this task. Reports system-level pairwise accuracy.

Evaluates the quality of multilingual word embeddings by measuring semantic similarity/relatedness and word categorization accuracy. It probes whether visual grounding improves cross-lingual semantic alignment and clustering of basic-level concepts. Use when the user wants to benchmark on WordSim353, MEN, RW, MTurk, simVerb, SimLex999, Battig, AP, BLESS, ESSLLI-a, ESSLLI-b, ESSLLI-c, Almarsoomi, MC30, Saif40, WordSim, or asks about evaluating this task. Reports Spearman correlation.

Evaluates how well multi-modal models learn word meanings and semantic relationships from limited data, comparing visual grounding against language-only baselines across word-relatedness, feature prediction, and POS tagging tasks. Use when the user wants to benchmark on SimLex-999, SimVerb-3500, Word-relatedness dataset (Bruni et al., 2012), or asks about evaluating this task. Reports human-likeness measure.

This benchmark evaluates a model's ability to perform word-level quality estimation across multiple language pairs, identifying whether translated words are correct ('OK') or incorrect ('BAD'), as well as detecting target gaps and source-side error triggers. It probes cross-lingual transfer and fine-grained alignment-aware error detection in machine translation. Use when the user wants to benchmark on WMT QE datasets (En-Zh, En-Cs, En-De, En-Ru, En-Lv, De-En), or asks about evaluating this ta...

Compute the WordErrorRate metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute WordErrorRate, or asks how to score with WordErrorRate.

Compute the WordInfoLost metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute WordInfoLost, or asks how to score with WordInfoLost.

Compute the WordInfoPreserved metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute WordInfoPreserved, or asks how to score with WordInfoPreserved.

Evaluates web agents' ability to perform complex, knowledge-worker tasks on enterprise UIs (ServiceNow) and standard web benchmarks. It probes multimodal browser observation processing, large DOM navigation, and action execution in interactive environments. Use when the user wants to benchmark on WorkArena, MiniWoB, WebGum Subset, or asks about evaluating this task. Reports success rate.

Evaluates whether automatically generated scientific workflow benchmarks accurately replicate the execution time and performance characteristics of real scientific workflows under varying hardware architectures and external memory loads. Use when the user wants to benchmark on Montage, 1000Genome, or asks about evaluating this task. Reports execution_time_ratio.

Evaluates the latency performance of AI workload allocation strategies across hierarchical cloud/edge/device computing environments for latency-sensitive medical ICU applications. It measures how effectively dynamic routing minimizes end-to-end response time when processing and transmission delays are factored in. Use when the user wants to benchmark on Edge AIBench ICU Applications (MIMIC-III derived), or asks about evaluating this task. Reports response time.

Evaluates AI models on work-domain recommendation and NLP tasks, primarily focusing on ranking and retrieval scenarios such as occupation-to-skill matching, candidate recommendation, and skill/job normalization. It tests cross-lingual and multilingual retrieval capabilities over standardized occupational ontologies like ESCO. Use when the user wants to benchmark on ESCO Occupation-to-Skill, ESCO Skill-to-Occupation, Job Title Sim., SkillMatch-1K, Query-Candidate, Project-Candidate, JobBERT, M...

Evaluates audio foundation models' cross-cultural generalization across diverse musical traditions (Western, Greek, Turkish, Indian) using multi-label tagging and few-shot learning. Probes whether pre-trained representations capture cultural musical knowledge without extensive adaptation. Use when the user wants to benchmark on Turkish-makam, Hindustani, Carnatic, MagnaTagATune, FMA-medium, Lyra, or asks about evaluating this task. Reports ROC-AUC.

Evaluates an agent's ability to automate desktop and web GUI tasks from arbitrary starting states. It probes robustness to dynamic initial conditions, contextual variations, and multi-step interaction planning in real-world software environments. Use when the user wants to benchmark on WorldGUI, or asks about evaluating this task. Reports Success Rate (SR).

Evaluates driving world models across five dimensions: generation quality, 3D/4D reconstruction coherence, action-following capability in closed-loop simulation, downstream perception task utility, and alignment with human preference. It probes geometric consistency, physical plausibility, and functional reliability of synthesized driving scenes. Use when the user wants to benchmark on WorldLens, or asks about evaluating this task. Reports Route Completion (%).

This benchmark evaluates interactive Image-to-Video world models by measuring their ability to generate temporally coherent videos in response to standardized action commands. It probes three core capabilities: visual fidelity, precise camera/object control alignment, and long-horizon world consistency across different perspectives and visual styles. Use when the user wants to benchmark on WorldMark Image Suite, or asks about evaluating this task. Reports Aesthetic Quality.

Evaluates multimodal video understanding and long-chain reasoning by requiring models to integrate visual, auditory, and external world knowledge to answer open-ended and multiple-choice questions. Use when the user wants to benchmark on WorldQA, or asks about evaluating this task. Reports GPT-4 open-ended score.

Evaluates multimodal large language models' ability to perform real-world omni-modal understanding by jointly processing tightly coupled audio and video inputs. It probes complex temporal reasoning, cross-modal integration, and fine-grained perception across diverse everyday scenarios. Use when the user wants to benchmark on WorldSense, or asks about evaluating this task. Reports accuracy.

Evaluates embodied world models on conditional video generation from an initial image and text instruction. It probes instruction understanding, long-horizon planning, physical/causal reasoning, and temporal consistency in robotic interaction scenarios. Use when the user wants to benchmark on WoWBench, or asks about evaluating this task. Reports Planning Score ($S_{plan}$), Overall Benchmark Score.

Evaluates a model's ability to perform sequential recommendation by predicting the next item a user will interact with based on their chronological interaction history. It probes the model's capacity to capture temporal dynamics and collaborative filtering signals while ranking items against a full candidate set. Use when the user wants to benchmark on MovieLens-1M*, Amazon-Beauty, Amazon-Sports, LastFM (HetRec 2011), or asks about evaluating this task. Reports HR@10.

Evaluates the ability of unsupervised and supervised metrics to identify word-level translation errors by comparing their continuous scores against human-annotated error spans and multi-annotator agreement rates. Use when the user has predictions and gold and needs to compute Average Precision (AP).

Evaluates a model's ability to disambiguate word senses for both common nouns and proper nouns exhibiting regular polysemy. It probes contextual understanding and the capacity to leverage structured sense glosses and dot-object type classes to select the correct meaning from a candidate inventory. Use when the user wants to benchmark on WSD dataset (CWN 2.0), RP dataset (Revised Mandarin Chinese Dictionary), or asks about evaluating this task. Reports accuracy.

Evaluates whole-slide image (WSI) classification performance using self-supervised patch representations and feature-space data augmentation. It probes how well distribution-guided representation learning captures discriminative histopathological patterns for diagnostic subtyping. Use when the user wants to benchmark on USTC-EGFR, TCGA-EGFR, TCGA-LUNG-3K, or asks about evaluating this task. Reports micro-average area under the curve (AUC).

Evaluates Word Sense Induction (WSI) by clustering contextualized word embeddings to predict sense assignments. It probes a model's ability to capture lexical polysemy and contextual meaning without supervised sense labels, using natural corpus distributions rather than artificially skewed benchmarks. Use when the user wants to benchmark on SemCor, or asks about evaluating this task. Reports F-B^3.

Evaluates speech recognition models on their ability to accurately transcribe spoken audio into text (WER) and characters (LER). It probes the effectiveness of unsupervised pre-training on raw audio for downstream acoustic modeling and decoding. Use when the user wants to benchmark on TIMIT, WSJ, or asks about evaluating this task. Reports WER.

This evaluation probes a model's ability to perform single-channel speech separation by learning discriminative time-frequency embeddings that group mixture components into distinct speaker clusters. It specifically tests generalization to unseen speakers and scaling to three-speaker mixtures without retraining. Use when the user wants to benchmark on WSJ0-based Speech Mixtures, or asks about evaluating this task. Reports SDR improvement (dB).

Evaluates machine learning classifiers for binary and multiclass intrusion detection in Wireless Sensor Networks (WSNs), specifically testing their robustness on imbalanced datasets and the impact of SMOTETomek resampling. Use when the user wants to benchmark on Large-scale WSN dataset, or asks about evaluating this task. Reports Accuracy.

Evaluates fine-grained spatial-temporal understanding and instance-aware video-to-text captioning in pedestrian-centric traffic scenarios. Probes a model's ability to accurately describe location, attention, behavior, and context of pedestrians and vehicles in complex traffic videos. Use when the user wants to benchmark on WTS, or asks about evaluating this task. Reports LLMScore.

Evaluates large language models' ability to understand and classify disruptive weather impacts from historical and modern newspaper articles, and to answer related questions by ranking relevant information. It specifically probes models' capacity to handle climate-related polysemy, extract nuanced societal responses, and correctly identify passages without weather impacts. Use when the user wants to benchmark on WXImpactBench, or asks about evaluating this task. Reports F1-score.

Evaluates an end-to-end navigation model's ability to predict robot dynamics and successfully navigate through structured and cluttered warehouse environments. It probes both open-loop trajectory and speed prediction accuracy, as well as closed-loop mission success, navigation efficiency, and motion smoothness in seen and out-of-distribution settings. Use when the user wants to benchmark on X-Mobility Warehouse Dataset, or asks about evaluating this task. Reports mission success rate (SR).

Evaluates the text rendering, text-to-image generation, and image understanding capabilities of a discrete autoregressive image generation model trained with reinforcement learning. It probes the model's ability to follow complex instructions, render long texts accurately, and generate high-fidelity images without relying on classifier-free guidance. Use when the user wants to benchmark on OneIG-Bench, LongText-Bench, DPG-Bench, GenEval, POPE, GQA, MMBench, SEEDBench-Img, DocVQA, OCRBench, or...

Evaluates multi-modal large language models on progressive clinical reasoning in ophthalmic diagnosis. It tests the model's ability to perform a six-stage diagnostic chain (from image quality assessment to clinical decision-making) while integrating cross-modality imaging data and calibrating its uncertainty. Use when the user wants to benchmark on X-PCR, or asks about evaluating this task. Reports Stage-Wise Accuracy (SWA).

This benchmark evaluates multilingual topic classification on social media tweets across four languages (English, Spanish, Japanese, Greek). It probes models' ability to generalize across languages and training regimes, including zero-shot, few-shot, monolingual, cross-lingual, and multilingual fine-tuning settings. Use when the user wants to benchmark on X-Topic, or asks about evaluating this task. Reports macro-F1.

Probes the spatial feature flow and patch-level influence in Vision Mamba models using classical control theory. It quantifies how input image patches drive hidden state dynamics across hierarchical layers, revealing domain-specific diagnostic feature extraction patterns. Use when the user wants to benchmark on CMMD, DermaMNIST, BloodMNIST, or asks about evaluating this task. Reports influence score.

Evaluates LLM-based agents' ability to comprehend multilingual shopping instructions and successfully navigate interactive web environments across 14 languages. Use when the user wants to benchmark on X-WebAgentBench, or asks about evaluating this task. Reports Task Score.

Evaluates the computational efficiency and explanation fidelity of an ROI-guided perturbation framework for medical image segmentation. It measures how effectively the method reduces computation while preserving segmentation accuracy and explanation quality compared to full occlusion baselines. Use when the user wants to benchmark on FLARE22, SAROS, CHAOS, or asks about evaluating this task. Reports Dice coefficient.