Research

Evaluates a model's ability to perform video summarization by predicting frame-level importance scores across visual, textual, and audio modalities. It probes the model's capacity for adaptive multimodal fusion and temporal dependency modeling to identify salient segments in long videos. Use when the user wants to benchmark on MoSu, Mr. HiSum, SumMe, TVSum, or asks about evaluating this task. Reports Kendall’s τ (kTau), Spearman’s ρ (sRho).

Evaluates large language models' ability to distinguish factually true statements from factually false and unverifiable ('neither') statements. It probes both prompt-based output probabilities and internal hidden activations to measure veracity classification accuracy and uncertainty quantification. Use when the user wants to benchmark on Trilemma of Truth Datasets, or asks about evaluating this task. Reports MCC.

This benchmark evaluates Turkish natural language understanding across multiple task types, including grammaticality judgment, sentiment analysis, paraphrase detection, and semantic textual similarity. It probes a model's ability to handle agglutinative morphology, culturally adapted expressions, and nuanced semantic equivalence in Turkish. Use when the user wants to benchmark on TrCoLA, TrSST-2, TrMRPC, TrSTS-B, or asks about evaluating this task. Reports accuracy.

Evaluates fact-checking systems across three sub-tasks: retrieving relevant evidence, verifying claim truthfulness, and generating natural language explanations. It specifically probes a model's Hotspot Perception Ability (HPA) by measuring how effectively it allocates reasoning effort and computational resources based on the real-world influence of trending claims. Use when the user wants to benchmark on TrendFact, or asks about evaluating this task. Reports F1-macro.

Evaluates single-object visual tracking performance in first-person vision videos, specifically testing robustness to object manipulation, occlusions, and dynamic interactions under real-time execution constraints. Use when the user wants to benchmark on TREK-150, or asks about evaluating this task. Reports SS, NPS, GSR.

Evaluates a model's ability to perform fine-grained tree species identification using multimodal Earth observation data, specifically leveraging temporal dynamics from optical and radar time series alongside high-resolution imagery. Use when the user wants to benchmark on TreeSatAI-TS, or asks about evaluating this task. Reports weighted F1.

This benchmark evaluates an LLM's ability to perform deep, structured scientific peer review by generating comprehensive reviews and actionable feedback comments. It probes the model's capacity for hierarchical question decomposition, context-aware analysis of long documents, and alignment with human reviewer judgments across multiple quality dimensions. Use when the user wants to benchmark on TreeReview Benchmark (ICLR-2024, NeurIPS-2023, Nature Communications), or asks about evaluating this...

TreeEval probes an LLM's ability to handle complex, adaptive reasoning through dynamically generated hierarchical questions. It evaluates how well a model's relative performance ranking aligns with established leaderboards like AlpacaEval2.0, while testing the framework's efficiency in distinguishing fine-grained capability differences without relying on static datasets. Use when the user wants to benchmark on TreeEval (Dynamic/Benchmark-Free), or asks about evaluating this task. Reports Spea...

Evaluates visual grounded reasoning by requiring models to precisely localize target objects in cluttered scenes and perform second-order reasoning about their interactions. It measures both the correctness of the final answer and the spatial accuracy of the traceable bounding box evidence. Use when the user wants to benchmark on TreeBench, or asks about evaluating this task. Reports Accuracy.

This evaluation protocol compares recursive tree-based neural models against recurrent sequence-based models across multiple NLP tasks. It probes whether syntactic tree structures are necessary for learning representations, particularly for tasks requiring long-distance dependency modeling or hierarchical composition. Use when the user wants to benchmark on Stanford Sentiment Treebank, Pang Sentiment Dataset, UMD-QA, SemEval-2010 Task 8, Discourse Parsing, or asks about evaluating this task. ...

Probes retrieval-augmented generation systems on complex, narrative-driven queries by decomposing information needs into sub-narratives. It evaluates document relevance based on sub-narrative coverage, measures response quality via strict vital recall of fully supported information nuggets, and assesses sentence-level factual grounding against cited documents. Use when the user wants to benchmark on MS MARCO V2.1, or asks about evaluating this task. Reports strict_vital_recall.

Evaluates the factual accuracy and content grounding of RAG-generated answers by checking for the presence of key factual claims (nuggets) extracted from source documents. It also measures answer length to assess the trade-off between conciseness and completeness in system outputs. Use when the user wants to benchmark on TREC 2024 RAG Track, or asks about evaluating this task. Reports V_strict.

Evaluates neural cross-language information retrieval systems on ad hoc, reranking, and monolingual tasks across Chinese, Persian, and Russian newswire collections. It measures how well models retrieve and rank relevant documents when queries are in English and documents are in other languages, or when queries are human-translated. Use when the user wants to benchmark on TREC 2022 NeuCLIR Collections, or asks about evaluating this task. Reports nDCG@10.

Evaluates retrieval systems on balancing topical relevance with intersectional fairness in Wikipedia article rankings. It probes static single-query ranking for coordinators and dynamic multi-query ranking for editors under fairness constraints across demographic attributes. Use when the user wants to benchmark on TREC 2022 Fair Ranking Track, or asks about evaluating this task. Reports M1, EE-L.

Evaluates information retrieval and re-ranking systems on their ability to balance document relevance with demographic fairness. It probes how well algorithms maintain ranking utility while ensuring equitable representation across inferred gender and country groups. Use when the user wants to benchmark on TREC 2020 Fairness Ranking Track dataset, or asks about evaluating this task. Reports utility.

Evaluates ad hoc information retrieval ranking methods on document and passage retrieval tasks using large-scale training data and human-labeled relevance judgments. It compares neural language models, neural networks, and traditional methods under blind single-shot conditions to assess ranking quality in the top-k results. Use when the user wants to benchmark on TREC 2020 Deep Learning Track, or asks about evaluating this task. Reports NDCG@10.

Evaluates the effectiveness of a learning-to-rank personalization approach for web search sessions by measuring relevance prediction quality across multiple years of session track data. It probes the model's ability to leverage historical query sequences, document rankings, and user click behavior to improve session-level relevance ranking. Use when the user wants to benchmark on TREC 2011-2014 Session Tracks, or asks about evaluating this task. Reports nDCG@k.

Evaluates real-time tweet summarization systems by measuring their ability to push relevant, non-redundant tweets within fixed temporal windows, while penalizing system latency and irrelevant outputs. Use when the user has predictions and gold and needs to compute Expected Gain (EG).

Evaluates the ability of LLM judges versus human annotators to assess sentence-level grounding (support) in RAG-generated answers. It measures how well models cite relevant passages and whether the cited text actually supports the generated claims. Use when the user wants to benchmark on TREC 2024 RAG Track, or asks about evaluating this task. Reports weighted precision.

Evaluates the effectiveness of query expansion methods for real-time microblog search by measuring how well ranked document lists match relevance judgments for short-form social media queries. It specifically probes the model's ability to handle vocabulary mismatch and temporal relevance in noisy, short-text retrieval scenarios. Use when the user wants to benchmark on TREC Microblog Track, or asks about evaluating this task. Reports MAP.

Evaluates information retrieval ranking methods on ad hoc and routing tasks across diverse text collections. It probes how well term weighting schemes capture semantic relevance and handle difficult or verbose queries compared to classic baselines like BM25 and TF-IDF. Use when the user wants to benchmark on TREC Collections and Topics, or asks about evaluating this task. Reports MAP.

This benchmark evaluates a system's ability to perform personalized conversational search by retrieving relevant passages and generating fluent, grounded responses. It specifically probes how well an agent can adapt its output to user-specific context encoded in a Personal Text Knowledge Base (PTKB) while maintaining provenance traceability. Use when the user wants to benchmark on TREC iKAT 2023, ClueWeb22-B Subset, or asks about evaluating this task. Reports groundedness.

Evaluates assistive RAG systems that support news trustworthiness assessment by generating investigative questions and context-rich reports. Probes the model's ability to identify critical aspects of source bias, motivation, and alternative viewpoints, and to synthesize attributed summaries that help readers evaluate credibility. Use when the user wants to benchmark on TREC DRAGUN 2025 Track, or asks about evaluating this task. Reports Kendall's τ.

Evaluates the reliability and best practices for using TREC Deep Learning test collections for ranking model evaluation. It probes whether researchers properly separate model selection from final evaluation and accounts for training variance. Use when the user wants to benchmark on TREC Deep Learning Track, or asks about evaluating this task. Reports NDCG@10.