Research

Evaluates sign language translation from video to spoken text. It probes the model's ability to handle long videos, large vocabularies, and high singleton rates by leveraging full-body and lip-reading visual features. Use when the user wants to benchmark on WMT 2022 Shared Task, PHOENIX 2014T, or asks about evaluating this task. Reports BLEU.

Evaluates machine translation quality estimation systems by predicting human judgments on translation adequacy and fluency (Direct Assessment) and classifying translation errors (CED). It probes the model's ability to correlate predicted scores with human ratings and accurately detect translation quality issues across multiple language pairs. Use when the user wants to benchmark on WMT 2021 Quality Estimation Shared Task datasets, or asks about evaluating this task. Reports Pearson's correlat...

Evaluates neural machine translation performance across news and biomedical domains for English-German and English-Russian language pairs. It probes the model's ability to handle domain-specific vocabulary, cross-lingual alignment, and translation quality under constrained data conditions typical of shared task tracks. Use when the user wants to benchmark on WMT21 News & Biomedical Shared Tasks, or asks about evaluating this task. Reports BLEU.

Evaluates the translation quality and inference speed of non-autoregressive versus autoregressive machine translation models on English-German news text. It probes the practical trade-offs between decoding latency and translation accuracy under realistic deployment conditions. Use when the user wants to benchmark on WMT21 News Translation, or asks about evaluating this task. Reports BLEU.

Evaluates Chinese-to-English machine translation performance on biomedical texts. It measures how well a neural MT system can translate domain-specific terminology and syntax while handling case, punctuation, and subword tokenization conventions. Use when the user wants to benchmark on WMT21 OK-aligned biomedical test set, or asks about evaluating this task. Reports BLEU.

Evaluates document-level and literary-domain machine translation quality, focusing on discourse-aware metrics like consistency, anaphora resolution, and term consistency across long-form Chinese web novels. The protocol combines automated n-gram and neural metrics with a structured human judgment framework to capture cross-sentence coherence and literary style. Use when the user wants to benchmark on WMT 2024 Discourse-Level Literary Translation Shared Task, or asks about evaluating this task...

Evaluates machine translation quality estimation across sentence-level scoring, word-level error detection, and fine-grained error span classification. It probes a model's ability to predict translation quality and localize specific errors without relying on human references during inference. Use when the user wants to benchmark on WMT2022 QE EN-DE dataset, WMT2022 Metric EN-DE dataset, WMT17/19/20 Post-editing EN-DE datasets, or asks about evaluating this task. Reports MCC.

Evaluates machine translation systems on discourse-level literary translation from Chinese to English, focusing on long-range context, cultural adaptation, and stylistic fidelity across entire web novels rather than isolated sentences. Use when the user wants to benchmark on WMT 2023 Discourse-Level Literary Translation Test Set, or asks about evaluating this task. Reports d-BLEU.

Evaluates neural machine translation systems across multiple language pairs (EN-DE, EN-CS, CS-EN, EN-RO, RO-EN, EN-RU, RU-EN) on news text. It measures translation quality using BLEU scores on held-out test sets to assess the impact of techniques like back-translation, ensembling, and subword segmentation. Use when the user wants to benchmark on WMT 2016 News Translation, or asks about evaluating this task. Reports BLEU.

Evaluates Automatic Post-Editing (APE) systems by measuring how well they correct machine-translated German sentences using monolingual and bilingual neural translation models combined via log-linear weighting. Use when the user wants to benchmark on WMT 2016 APE Shared Task Development Set, or asks about evaluating this task. Reports TER.

Evaluates machine translation quality by scoring system-generated English translations against human reference sentences and expert MQM scores. It measures how well automatic metrics correlate with human judgments across different translation systems. Use when the user wants to benchmark on WMT20 ZH-EN, or asks about evaluating this task. Reports COMET.

Evaluates machine translation quality by scoring system-generated German translations against human reference sentences and expert MQM scores. It measures how well automatic metrics correlate with human judgments across different translation systems. Use when the user wants to benchmark on WMT20 EN-DE, or asks about evaluating this task. Reports COMET.

Evaluates machine translation quality between similar languages (Czech to Polish) using a multi-encoder transformer trained on out-of-domain data filtered by cross-entropy differences. Probes the model's ability to adapt to low-resource similar language pairs via domain adaptation and data selection. Use when the user wants to benchmark on WMT19 SLT Shared Task dataset, or asks about evaluating this task. Reports BLEU.

Evaluates an online learning framework's ability to dynamically identify the highest-quality machine translation systems from an ensemble using minimal human feedback, and measures the sample efficiency (number of human assessments needed) to converge to the official top-performing systems. Use when the user wants to benchmark on WMT'19 News Translation, or asks about evaluating this task. Reports convergence_to_top3.

Evaluates the effectiveness of automatic web data selection for domain-specific machine translation in the news domain. It probes whether a document-level topic classifier can filter noisy parallel data to improve MT system performance compared to state-of-the-art baselines on the WMT-18 benchmark. Use when the user wants to benchmark on WMT-18 News Shared Task, or asks about evaluating this task. Reports BLEU.

Evaluates how well a model's predicted quality scores correlate with human judgments on machine translation output. It probes semantic equivalence and paraphrase detection capabilities in the context of MT evaluation. Use when the user wants to benchmark on WMT17, or asks about evaluating this task. Reports Pearson |r|.

Evaluates the training efficiency, inference speed, and translation quality of neural machine translation systems on standard WMT17 benchmarks. It probes the trade-offs between model architecture, hardware acceleration (FP16/INT8), and batching strategies. Use when the user wants to benchmark on WMT17 English-German, WMT17 Russian-English, or asks about evaluating this task. Reports BLEU.

Evaluates neural machine translation systems across multiple language pairs in news and biomedical domains. Probes translation quality, domain adaptation, and system combination techniques like ensembling and reranking on held-out parallel test sets. Use when the user wants to benchmark on WMT17 News Task, HimL Biomedical Task, or asks about evaluating this task. Reports BLEU.

Evaluates the effectiveness of an RNN Encoder-Decoder architecture for statistical machine translation on English-to-French tasks. It measures how well the model learns phrase representations and improves translation quality over a traditional phrase-based baseline system. Use when the user wants to benchmark on WMT'14 English/French, or asks about evaluating this task. Reports BLEU.

Evaluates the capability of sign language to text translation systems on low-resource datasets. It measures how accurately a model can convert 3D pose sequences of sign language into corresponding spoken language text. Use when the user wants to benchmark on FocusNews, SRF, or asks about evaluating this task. Reports BLEU.

This protocol evaluates the machine translation quality of large language models across multiple language pairs. It measures translation accuracy and fluency by comparing model outputs against gold references and state-of-the-art baselines using neural quality estimation metrics. The benchmark probes the model's ability to generalize across diverse language directions and avoid generating near-perfect but flawed translations. Use when the user wants to benchmark on WMT'21 Test Set, WMT'22 Tes...

Evaluates the quality of an unsupervised web-mined parallel corpus by training neural machine translation models on it and measuring translation performance on standard WMT test sets. It probes whether mined pseudo-parallel data can effectively substitute for human-labeled data in both supervised and unsupervised MT training pipelines. Use when the user wants to benchmark on WMT2014 test set, WMT2016 test set, or asks about evaluating this task. Reports BELU.

Evaluates a model's ability to perform Automatic Post-Editing (APE) by correcting machine-translated German sentences using the original source text. It probes error detection, grammatical correction, and word-copying capabilities in a multi-source sequence-to-sequence setting. Use when the user wants to benchmark on WMT APE, or asks about evaluating this task. Reports case-sensitive BLEU.

Evaluates large language models' knowledge of hazardous topics in biosecurity, cybersecurity, and chemical security, as well as their general knowledge and fluency. It serves as a proxy for measuring dual-use risk and benchmarking unlearning methods. Use when the user wants to benchmark on WMDP, or asks about evaluating this task. Reports WMDP.