Evetar Eval

---
name: evetar-eval
description: EveTAR evaluates Arabic information retrieval systems across four tasks: event detection, ad-hoc search, tweet timeline generation, and real-time summarization. It probes a model's ability to retrieve, cluster, and summarize relevant Arabic tweets in response to event-driven queries or topics. Use when the user wants to benchmark on EveTAR, or asks about evaluating this task. Reports recall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1708.05517
  bibtex_key: hasanain2017evetar
  confidence: high
---

# evetar-eval

> EveTAR: Building a Large-Scale Multi-Task Test Collection over Arabic Tweets — Hasanain et al. (2017) (arXiv:1708.05517, 2017)

## What this evaluates

EveTAR evaluates Arabic information retrieval systems across four tasks: event detection, ad-hoc search, tweet timeline generation, and real-time summarization. It probes a model's ability to retrieve, cluster, and summarize relevant Arabic tweets in response to event-driven queries or topics.

## Datasets

- **EveTAR** — total 355000000; splits: EveTAR-F (-1), EveTAR-S (-1), EveTAR-S.m (-1), EveTAR-S.d (-1), EveTAR-Q (-1)

## Metrics

- `recall` **(primary)** — range: [0, 1]
  - recall = #covered events / #reference events. A reference event is considered covered if at least 50% of the tweets in the detected event belong to that reference event.
- `MAP` — range: [0, 1]
  - Mean Average Precision averaged over all 50 topics.
- `P@30` — range: [0, 1]
  - Precision at rank 30 averaged over all 50 topics.
- `wF1` — range: [0, 1]
  - Weighted F1 score used as the official evaluation measure for tweet timeline generation in TREC 2014.
- `EG-1` — range: [0, 1]
  - Expected Gain metric used for real-time summarization evaluation.
- `nCG-1` — range: [0, 1]
  - Normalized Cumulative Gain metric used for real-time summarization evaluation.

## Input / output format

**Input**: For each task, the system receives a collection of Arabic tweets. Event detection receives the full tweet collection. Ad-hoc search and TTG receive a query/topic and the tweet collection. RTS receives a topic title and a stream of tweets.

**Output**: Event detection: list of detected events (each a list of tweets). Ad-hoc search: ranked list of tweets. TTG: timeline of summarized tweets. RTS: set of pushed tweets or silent decision.

## Scoring recipe

```python
def compute_recall(detected_events, reference_events):
    covered = 0
    for ref_event in reference_events:
        ref_tweets = set(ref_event.tweets)
        for det_event in detected_events:
            det_tweets = set(det_event.tweets)
            if len(ref_tweets & det_tweets) >= 0.5 * len(ref_tweets):
                covered += 1
                break
    return covered / len(reference_events)
```

## Common pitfalls

- Precision cannot be automatically computed for event detection because the complete set of possible significant events in the collection is unknown.
- Task performance is highly sensitive to language mismatch; e.g., RTS systems using Modern Standard Arabic topic titles on dialectal Arabic tweets yield artificially low precision.
- The Generalizability Coefficient (GC) measures ranking reliability for a specific set of systems, not the absolute quality of the test collection.

## Evidence (verbatim from paper)

> To automatically evaluate the algorithms, we adopted Petrovic's (Petrovič 2013) approach which measures effectiveness using recall: recall = #covered events / #reference events where # reference events is the number of events in EveTAR and # covered events is the number of reference events covered by the algorithm's detected events (where an event is represented by a list of tweets).

## Citation

```bibtex
@misc{hasanain2017evetar,
  title={EveTAR: Building a Large-Scale Multi-Task Test Collection over Arabic Tweets},
  author={Hasanain et al. (2017)},
  year={2017},
  note={arXiv:1708.05517}
}
```

- arXiv: 1708.05517