Semeval Question Relevancy Eval

---
name: semeval-question-relevancy-eval
description: Evaluates a model's ability to rank question-answer pairs by relevance. It probes the system's capacity to understand semantic similarity and perform information retrieval tasks using language-independent features and multi-task learning. Use when the user wants to benchmark on SemEval-2016 Task 3, or asks about evaluating this task. Reports MAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1808.08836
  bibtex_key: gonzalezgarduno2018strong
  confidence: medium
---

# semeval-question-relevancy-eval

> A strong baseline for question relevancy ranking — González-Garduño et al. (2018) (arXiv:1808.08836, 2018)

## What this evaluates

Evaluates a model's ability to rank question-answer pairs by relevance. It probes the system's capacity to understand semantic similarity and perform information retrieval tasks using language-independent features and multi-task learning.

## Datasets

- **SemEval-2016 Task 3** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `MAP` **(primary)** — range: [0, 100]
  - Mean Average Precision; standard information retrieval metric averaging precision at relevant document ranks across queries.
- `Accuracy` — range: [0, 100]
  - Accuracy; fraction of correctly classified or correctly ranked pairs.

## Input / output format

**Input**: Pairs of questions (or question-answer pairs) to be evaluated for relevance.

**Output**: Ranked list of pairs by relevance score, or binary/multi-class relevance labels.

## Scoring recipe

```python
# MAP calculation
ap_scores = []
for query in queries:
    relevant_docs = gold_relevance[query]
    ranked_docs = model_rank(query)
    hits = 0
    sum_prec = 0
    for i, doc in enumerate(ranked_docs):
        if doc in relevant_docs:
            hits += 1
            sum_prec += hits / (i + 1)
    ap_scores.append(sum_prec / len(relevant_docs))
map_score = sum(ap_scores) / len(ap_scores)
# ACC calculation
acc_score = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
```

## Common pitfalls

- Confusing the primary ranking metric (MAP) with classification accuracy (Accuracy), as both are reported but MAP is the official SemEval metric.
- Assuming full auxiliary datasets (FNC, MultiNLI) are used for training; the paper explicitly samples them to match the size of SemEval's train/dev/test sets.
- Overlooking that the model relies on simple language-independent distance metrics (cosine, Euclidean, Jaccard) rather than complex deep architectures.

## Evidence (verbatim from paper)

> We present the official metric from the SemEval task, as well as other common metrics. For the SemEval-16 data, our multitask MLP architecture with a question-answer auxiliary task performed best on all metrics, except accuracy, where the multi-task MLP using all auxiliary tasks performed best. We outperform the winning systems of both the SemEval 2016 and 2017 campaigns.

## Citation

```bibtex
@misc{gonzalezgarduno2018strong,
  title={A strong baseline for question relevancy ranking},
  author={González-Garduño et al. (2018)},
  year={2018},
  note={arXiv:1808.08836}
}
```

- arXiv: 1808.08836