Research

Evaluates zero-shot language-queried speech enhancement by isolating clean speech from noisy backgrounds. The benchmark tests the model's ability to enhance speech using a fixed text query. Use when the user wants to benchmark on Voicebank-DEMAND, or asks about evaluating this task. Reports PESQ.

Evaluates AI voice assistants across listening, speaking, and viewing capabilities. It probes audio understanding, multi-turn dialogue generation, role-play imitation, and multimodal vision-audio integration, measuring both content accuracy and speech naturalness. Use when the user wants to benchmark on VoiceAssistant-Eval, or asks about evaluating this task. Reports Final Task Score.

Evaluates speech recognition accuracy and latency trade-offs for streaming vs. non-streaming decoding on a proprietary voice-search dataset. It probes the model's ability to maintain low word error rate while minimizing output delay and computational overhead. Use when the user wants to benchmark on Voice Search, or asks about evaluating this task. Reports WER.

Evaluates automatic speech recognition (ASR) systems on real-world, unscripted telephonic conversations across 15 Indian languages. It probes geographic, demographic, and audio quality disparities in model performance, particularly focusing on code-mixed speech and natural orthographic variations. Use when the user wants to benchmark on Voice of India, or asks about evaluating this task. Reports Word Error Rate (WER).

This evaluation probes auditory self-recognition boundaries by measuring how much AI voice morphing a participant can tolerate before they stop recognizing their own voice. It assesses perceptual thresholds, decision latency, and the influence of acoustic embedding distances and demographic factors on voice identity perception. Use when the user wants to benchmark on VoiceMorph Experimental Dataset, or asks about evaluating this task. Reports lowess_T.

Evaluates the privacy guarantee (voice-indistinguishability) and utility of perturbed speech data. It measures how effectively a sanitization framework hides speaker identity while preserving speech recognition accuracy and perceptual naturalness. Use when the user has predictions and gold and needs to compute ACC (Speaker Verification Accuracy).

Evaluates non-parallel voice conversion quality by measuring spectral distortion, pitch accuracy, voicing correctness, duration modification capability, and subjective naturalness/speaker similarity. Use when the user wants to benchmark on VCTK, CMU ARCTIC, or asks about evaluating this task. Reports MCD.

Evaluates how commercial voice cloning systems preserve speaker identity and speech intelligibility for standard versus accented Mandarin speakers. It probes the alignment between acoustic embedding distances and human perceptual judgments of similarity and intelligibility across accent conditions. Use when the user wants to benchmark on Standard and Accented Mandarin Speech Dataset, or asks about evaluating this task. Reports Intelligibility gain score.

This benchmark evaluates the quality of commercial voice AI testing platforms across two independent dimensions: simulation quality (how realistically platforms generate test conversations) and evaluation accuracy (how accurately platforms assess conversation quality against human ground truth). It probes whether automated testing systems can reliably replace human quality assurance in high-stakes voice AI deployments. Use when the user wants to benchmark on Custom Voice AI Testing Benchmark,...

Evaluates a model's ability to separate vocal and accompaniment tracks from mixed music audio. It probes long-term dependency modeling and pattern repetition exploitation in audio source separation. Use when the user wants to benchmark on DSD100, MedleyDB, CCMixer, or asks about evaluating this task. Reports SDR.

Evaluates the intrinsic geometric alignment and zero-shot content identity of frozen audio embeddings across diverse single-source audio corpora. It measures how well models can retrieve semantically similar audio clips without task-specific fine-tuning, highlighting generalization gaps on low-resource or out-of-distribution speech. Use when the user wants to benchmark on VocSim, or asks about evaluating this task. Reports GSR.

Evaluates the audio synthesis quality, computational efficiency, and speaker generalization of neural vocoders across autoregressive, GAN-based, and diffusion-based architectures. It probes how well models preserve waveform fidelity and spectrogram structure while balancing inference speed and training complexity. Use when the user wants to benchmark on LJ Speech, LibriTTS, VCTK, or asks about evaluating this task. Reports MOS.

Evaluates a latent diffusion purification model's ability to remove adversarial perturbations from voiceprint defenses while preserving speaker identity and perceptual quality. It measures how effectively the purifier restores speaker verification scores and maintains speech naturalness and intelligibility for downstream voice cloning tasks. Use when the user wants to benchmark on LibriSpeech, VCTK, or asks about evaluating this task. Reports ARR.

Evaluates Mandarin speech-to-speech conversational agents across semantic understanding, acoustic quality, dialogue management, and robustness. It probes capabilities like cultural context adaptation, emotional empathy, instruction following, and handling of code-switching or noisy inputs. Use when the user wants to benchmark on VocalBench-zh, or asks about evaluating this task. Reports Accuracy.

This benchmark evaluates end-to-end speech interaction models across semantic understanding, acoustic quality, conversational fluency, and robustness to noisy or diverse inputs. It probes the model's ability to generate natural, emotionally expressive speech while accurately following instructions and maintaining safety alignment. The evaluation also measures computational efficiency and latency to assess real-time usability. Use when the user wants to benchmark on VocalBench, or asks about e...

Evaluates the quality of Vietnamese text embeddings across six standard information retrieval and NLP tasks. It probes a model's ability to capture semantic similarity, perform document retrieval, classify text, cluster documents, and rank relevant passages in Vietnamese. Use when the user wants to benchmark on VN-MTEB, or asks about evaluating this task. Reports Average Task Score.

Evaluates how well text-to-video models generate videos that align with human perceptual preferences across five motion dimensions: object integrity, motion smoothness, commonsense adherence, perceptible amplitude, and temporal coherence. Use when the user wants to benchmark on MMPG-set, or asks about evaluating this task. Reports Spearman correlation.

Evaluates Vietnamese natural language understanding across five diverse tasks including machine reading comprehension, natural language inference, emotion recognition, hate speech detection, and part-of-speech tagging. It assesses a model's ability to comprehend text, reason over sentence pairs, classify emotions and hate speech, and perform syntactic analysis in Vietnamese. Use when the user wants to benchmark on UIT-ViQuAD 2.0, ViNLI, VSMEC, ViHOS, NIIVTB POS, or asks about evaluating this ...

Evaluates the safety alignment of multimodal large language models (MLLMs) by testing their ability to correctly identify and appropriately respond to unsafe image-text pairs. It specifically probes how well models handle Visual Safety Information Leakage (VSIL), where harmful content might be implicitly revealed in the textual query rather than the image. Use when the user wants to benchmark on VLSBench, or asks about evaluating this task. Reports safety rate (%).

Evaluates an agent's ability to decompose coarse-grained natural language navigation instructions into executable subtasks and navigate through simulated environments to reach target locations or interact with objects. It probes task planning, visual-language grounding, and dynamic error recovery in continuous or discrete navigation spaces. Use when the user wants to benchmark on R2R, REVERIE, ALFRED, or asks about evaluating this task. Reports Success Rate (SR).

Evaluates an agent's ability to follow natural language instructions to navigate to a target location in a continuous 3D environment. It probes low-level action control, obstacle avoidance, and spatial reasoning without relying on a pre-defined graph topology or oracle localization. Use when the user wants to benchmark on VLN-CE, or asks about evaluating this task. Reports SR, SPL.

This benchmark evaluates a robot agent's ability to execute 6D manipulation tasks guided by natural language instructions and visual observations. It probes compositional reasoning, object localization, and precise pose estimation in both seen and unseen object settings. Use when the user wants to benchmark on VLMbench, or asks about evaluating this task. Reports success rate.

Evaluates the safety and generalization capabilities of vision-language models by measuring their ability to redirect unsafe content to safe alternatives, maintain zero-shot classification accuracy, and generate safe text/images from unsafe prompts or inputs. Use when the user wants to benchmark on ViSU, NSFWCaps, I2P, NudeNet/SMID/NSFW URLs, Zero-shot Benchmarks (ImageNet variants, Caltech101, Oxford Pets, Flowers102, Stanford Cars, UCF101, DTD), or asks about evaluating this task. Reports %...

Evaluates vision-language models on general visual understanding, spatial/relational reasoning, and specifically interactional reasoning in dynamic scenes using a suite of standard VQA and scene understanding benchmarks. Use when the user wants to benchmark on VQAv2, VizWiz, TextVQA, GQA, VSR, RealWorldQA, MMT-Bench, SEEDBench, A-Bench, or asks about evaluating this task. Reports accuracy.