Multiclaim Eval

---
name: multiclaim-eval
description: Tests the ability of retrieval models to rank previously fact-checked claims relevant to a given social media post. It probes crosslingual and monolingual claim retrieval capabilities, evaluating how well models handle multilingual text, varying post lengths, and different fact-check ratings. Use when the user wants to benchmark on MultiClaim, or asks about evaluating this task. Reports S@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.07991
  bibtex_key: pikuliak2023multiclaim
  confidence: high
---

# multiclaim-eval

> Multilingual Previously Fact-Checked Claim Retrieval — Pikuliak et al. (2023) (arXiv:2305.07991, 2023)

## What this evaluates

Tests the ability of retrieval models to rank previously fact-checked claims relevant to a given social media post. It probes crosslingual and monolingual claim retrieval capabilities, evaluating how well models handle multilingual text, varying post lengths, and different fact-check ratings.

## Datasets

- **MultiClaim** — total 31000; splits: test (31000); repo https://github.com/kinit-sk/multiclaim

## Metrics

- `S@K` **(primary)** — range: [0, 1]
  - Percentage of test pairs where the correct fact-check appears in the top K ranked results.

## Input / output format

**Input**: Query: social media post (main text + OCR transcript). Candidate pool: fact-check claims (text only).

**Output**: Ranked list of fact-check claims sorted by relevance score.

## Scoring recipe

```python
def success_at_k(predictions, gold, k=10):
    hits = 0
    for gold_id in gold:
        if gold_id in predictions[:k]:
            hits += 1
    return hits / len(gold)
```

## Common pitfalls

- Same language bias (SLB) can artificially inflate monolingual scores if the full multilingual fact-check pool is used without restricting to the post's language.
- BM25 performance is highly sensitive to script type (e.g., fails on Thai/Myanmar due to scriptio continua) and candidate pool size.
- Crosslingual evaluation requires searching the entire multilingual fact-check index, whereas monolingual evaluation restricts the search to the post's language pool.

## Evidence (verbatim from paper)

> We evaluate the performance based on the rank of the desired fact-checks by using success-at-K (S@K) as the main evaluation metric. We define as the percentage of pairs when the desired fact-check ends up in the top K.

## Citation

```bibtex
@misc{pikuliak2023multiclaim,
  title={Multilingual Previously Fact-Checked Claim Retrieval},
  author={Pikuliak et al. (2023)},
  year={2023},
  note={arXiv:2305.07991}
}
```

- arXiv: 2305.07991