Qilin Eval

---
name: qilin-eval
description: Evaluates multimodal information retrieval systems across search, recommendation, and deep query answering (DQA) tasks using real-world APP-level user sessions. It probes a model's ability to rank heterogeneous content (text, images, videos) and generate accurate answers augmented by retrieved documents. Use when the user wants to benchmark on Qilin, or asks about evaluating this task. Reports MRR@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.00501
  bibtex_key: chen2025qilin
  confidence: high
---

# qilin-eval

> Qilin: A Multimodal Information Retrieval Dataset with APP-level User Sessions — Jia Chen et al. (arXiv:2503.00501, 2025)

## What this evaluates

Evaluates multimodal information retrieval systems across search, recommendation, and deep query answering (DQA) tasks using real-world APP-level user sessions. It probes a model's ability to rank heterogeneous content (text, images, videos) and generate accurate answers augmented by retrieved documents.

## Datasets

- **Qilin** — total ?; splits: train (-1), test (-1); repo https://github.com/RED-Search/Qilin

## Metrics

- `MRR@10` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank at K: 1/N ∑(I(r_i ≤ K)/r_i), where N is the number of test instances and r_i is the rank of the first positive result.
- `MRR@100` — range: [0, 1]
  - Same as MRR@10 but evaluated at K=100.
- `MAP@10` — range: [0, 1]
  - Mean Average Precision at K: 1/N ∑(1/R_i ∑ P_i,k · I(rel_i,k=1)), where R_i is total relevant results, P_i,k is precision at rank k, and rel_i,k is binary relevance (clicks).
- `MAP@100` — range: [0, 1]
  - Same as MAP@10 but evaluated at K=100.
- `ROUGE-L` — range: [0, 1]
  - Longest Common Subsequence-based F1 score measuring syntactic overlap between generated and reference answers.
- `BERTScore (F1)` — range: [0, 1]
  - Semantic similarity score computed using contextual embeddings from BERT, reporting the F1 harmonic mean of precision and recall.

## Input / output format

**Input**: Query text, note title, note content, and optionally cover images. For recommendation, concatenated titles of recently clicked notes act as pseudo-query. For DQA, query plus retrieved reference documents (or none).

**Output**: Ranked list of notes (search/recommendation) or generated text answer (DQA).

## Scoring recipe

```python
def compute_mrr(ranked_rels, K):
    for rank, rel in enumerate(ranked_rels, 1):
        if rel == 1: return 1.0 / rank
    return 0.0
def compute_map(ranked_rels, K):
    prec, rel_cnt = [], 0
    for k, rel in enumerate(ranked_rels[:K], 1):
        if rel == 1:
            rel_cnt += 1
            prec.append(rel_cnt / k)
    return np.mean(prec) if prec else 0.0
mrr_k = np.mean([compute_mrr(r, K) for r in test_rels])
map_k = np.mean([compute_map(r, K) for r in test_rels])
rouge_l = rouge_l_score(pred, gold)
bertscore_f1 = bertscore_score(pred, gold, lang='auto')
```

## Common pitfalls

- Time-based split (11:1 ratio by hour) may cause data leakage if future sessions influence past training.
- Using clicks as binary relevance labels ignores implicit negative feedback and may introduce noise.
- DQA evaluation uses zero-shot RAG without fine-tuning, which may not reflect real-world deployment performance.

## Evidence (verbatim from paper)

> For evaluation metrics, we choose Mean Reciprocal Rank (MRR) and Mean Average Precision (MAP), which can be formulated as: MRR@K = 1/N ∑(I(r_i ≤ K)/r_i), MAP@K = 1/N ∑(1/R_i ∑ P_i,k · I(rel_i,k=1)) where N denotes the total number of testing instances, r_i represents the rank of the first positive result in the reranked list. As for rel_i,k, we use clicks as binary relevance labels for simplicity.

## Citation

```bibtex
@misc{chen2025qilin,
  title={Qilin: A Multimodal Information Retrieval Dataset with APP-level User Sessions},
  author={Jia Chen et al.},
  year={2025},
  note={arXiv:2503.00501}
}
```

- arXiv: 2503.00501