Research

Evaluates text-to-video diffusion models on visual quality and semantic alignment with input prompts. It measures intra-frame fidelity, aesthetic appeal, and inter-frame temporal consistency using automated benchmarks and human-preference predictors. Use when the user wants to benchmark on VBench, or asks about evaluating this task. Reports VBench.

Evaluates a model's ability to track arbitrary visual instances across complex, unstructured real-world videos without assuming motion continuity or fixed categories. It measures both local search accuracy and global robustness against challenges like occlusion, fast motion, and scene transitions. Use when the user wants to benchmark on VideoCube, or asks about evaluating this task. Reports PRE.

Evaluates a model's ability to learn transferable, long-horizon action dynamics from real-world videos and generate coherent task execution sequences across different environments and robotic setups. Use when the user wants to benchmark on Video-CraftBench, CALVIN, or asks about evaluating this task. Reports Sequential Success Rate (%).

Evaluates whether LLMs and MLLMs can accurately extract stock tickers, identify explicit investment actions, and quantify human conviction levels from financial influencer videos and transcripts. It probes multimodal reasoning, financial domain understanding, and the ability to filter out noisy or promotional content. Use when the user wants to benchmark on VideoConviction, or asks about evaluating this task. Reports F1 score.

Evaluates large multimodal models' ability to perceive and judge video aesthetics across visual form, style, and affectiveness dimensions. It tests performance on diverse video sources (UGC, AIGC, RGC, compression, gaming) using multiple-choice, true/false, and open-ended questions. Use when the user wants to benchmark on VideoAesBench, or asks about evaluating this task. Reports accuracy.

This evaluation protocol assesses a model's ability to generate high-fidelity, diverse instrumental music that is semantically and temporally aligned with a given 10-second video and optional fine-grained text prompt. It probes audio quality, distributional fidelity, generative diversity, and cross-modal alignment using both automated perceptual metrics and human/LLM preference judgments. Use when the user wants to benchmark on ReelBench, LORIS, V2MBench, or asks about evaluating this task. R...

Evaluates fine-grained video understanding, spatio-temporal reasoning, and hallucination mitigation in multimodal large language models by testing their ability to locate key visual cues and answer questions across diverse video benchmarks. Use when the user wants to benchmark on VSI-Bench, VideoMMMU, MMVU, MVBench, TempCompass, VideoMME, VideoHallucer, or asks about evaluating this task. Reports Accuracy.

Evaluates a model's ability to generate temporally consistent, animated 3D meshes from input videos. It probes per-frame geometric reconstruction accuracy, overall 4D sequence fidelity, and motion transfer quality while maintaining topology consistency across frames. Use when the user wants to benchmark on Objaverse, Consistent4D, DAVIS, or asks about evaluating this task. Reports CD-3D.

Evaluates video large language models on their ability to understand complex visual narratives and answer questions correctly. It specifically probes robustness by testing model performance on naturally adversarial or misleading variations of the same video question. Use when the user wants to benchmark on Video Thinking Test, or asks about evaluating this task. Reports Correctness score (accuracy).

Evaluates open-vocabulary action recognition by testing a model's ability to generalize to unseen action categories and cross-dataset distributions. It probes fine-grained video understanding and cross-modal reasoning capabilities under base-to-novel and cross-dataset generalization settings. Use when the user wants to benchmark on UCF-101, HMDB-51, Kinetics-400, Kinetics-600, Something-Something V2, or asks about evaluating this task. Reports Top-1 accuracy.

This benchmark probes the ability of video-language models and humans to distinguish real ASMR videos from AI-generated ones, evaluating perceptual realism and audio-visual consistency. It also measures how effectively video generation models can deceive video understanding models by producing indistinguishable synthetic content. Use when the user wants to benchmark on Video Reality Test, or asks about evaluating this task. Reports accuracy.

Evaluates video large multimodal models on question-answering tasks across multiple benchmark datasets. Probes the model's ability to understand video content, generate factually accurate long-form responses, and align with language model-derived preferences using direct preference optimization. Use when the user wants to benchmark on MSVD-QA, MSRVTT-QA, TGIF-QA, ActivityNet-QA, VIDAL-QA, WebVid-QA, SSV2-QA, or asks about evaluating this task. Reports accuracy.

Evaluates the ability of generative models to perform long-horizon open-loop video prediction. It probes how well models maintain temporal consistency, preserve object identities, and adapt to varying scene dynamics across diverse visual domains. Use when the user wants to benchmark on MineRL Navigate, KTH Action, GQN Mazes, Moving MNIST, or asks about evaluating this task. Reports FVD.

Evaluates the ability of text-to-video models to generate physically plausible content by testing adherence to real-world action-centric physical rules, such as conservation of mass/momentum and gravity. It probes whether models understand fundamental physical commonsense beyond superficial motion or visual aesthetics. Use when the user wants to benchmark on VideoPhy-2, or asks about evaluating this task. Reports joint_performance.

Evaluates a model's ability to generate spatially and temporally consistent video content outside the original frame boundaries (video outpainting), while preserving source structure and visual realism. Use when the user wants to benchmark on DAVIS 2017, YouTube-VOS, or asks about evaluating this task. Reports FVD.

This protocol audits video understanding benchmarks to measure genuine spatio-temporal reasoning versus shortcut reliance. It filters out samples solvable without video context and evaluates models under diagnostic conditions (e.g., blind, audio-only, center-frame) to quantify performance degradation and dependency on actual video content. Use when the user wants to benchmark on EgoSchema, ImplicitQA, VSI-Bench, TVBench, VCR-Bench, RTV-Bench, Video-Holmes, MINERVA, MMR-V, VideoMME, MVBench, L...

Evaluates long-horizon video understanding and reasoning capabilities of multimodal models on extended video sequences. Use when the user wants to benchmark on Video-MME, or asks about evaluating this task. Reports accuracy.

Evaluates a model's ability to segment moving and unknown objects in autonomous driving videos without relying on a closed set of known classes. It probes open-set and motion-based instance segmentation capabilities under varying data distributions and synthetic scenarios. Use when the user wants to benchmark on Cityscapes-VPS, KITTI-MOTS, Carla, or asks about evaluating this task. Reports CAQ, CA-IoU.

Evaluates fine-grained audiovisual captioning quality, attribute-level instruction following, and downstream reasoning capabilities like QA and temporal grounding. Use when the user wants to benchmark on video-SALMONN-2, UGC-VideoCap, VDC, VidCapBench-AE, Daily-Omni, World-Sense, Charades-STA, or asks about evaluating this task. Reports accuracy.

Evaluates video generation models across two core dimensions: video-condition alignment (how well the generated video matches the text prompt in terms of objects, actions, colors, scenes, and overall consistency) and video quality (technical fidelity, aesthetics, temporal consistency, and motion quality). Use when the user wants to benchmark on Video-Bench, or asks about evaluating this task. Reports Video-text Consistency.

Evaluates long-video understanding and agentic retrieval capabilities by testing models on temporal grounding, summarization, reasoning, and event causality across ultra-long video streams. Use when the user wants to benchmark on LVBench, VideoMME-Long, Ava-100, or asks about evaluating this task. Reports accuracy.

Evaluates a model's ability to recognize and classify human actions in video clips by predicting action categories from sampled frames. It probes temporal dynamics modeling and spatial feature extraction capabilities in video understanding tasks. Use when the user wants to benchmark on Kinetics-400, Something-Something-V2, Epic-Kitchens-100, or asks about evaluating this task. Reports Top-1 accuracy.

Egocentric video understanding for embodied AI, probing capabilities in video question-answering, hierarchical task planning, visual grounding, and reward modeling. It evaluates how well multimodal models comprehend first-person, action-oriented video contexts required for robotic interaction. Use when the user wants to benchmark on VidEgoThink, or asks about evaluating this task. Reports accuracy.

Evaluates a model's ability to assess danger levels in videos by identifying risk elements, understanding context, and assigning severity scores. It probes multimodal perception and risk reasoning capabilities. Use when the user wants to benchmark on ViDAS, or asks about evaluating this task. Reports MSE.