Semeval2017task4 Eval

---
name: semeval2017task4-eval
description: Evaluates the ability of models to classify sentiment in social media posts (tweets) across different languages (English and Arabic) and granularities (overall polarity, topic-specific polarity, and ordinal scales). Use when the user wants to benchmark on SemEval-2017 Task 4, or asks about evaluating this task. Reports macro-average recall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1912.00741
  bibtex_key: rosenthal2017semeval
  confidence: high
---

# semeval2017task4-eval

> SemEval-2017 Task 4: Sentiment Analysis in Twitter — Rosenthal et al. (2017) (arXiv:1912.00741, 2017)

## What this evaluates

Evaluates the ability of models to classify sentiment in social media posts (tweets) across different languages (English and Arabic) and granularities (overall polarity, topic-specific polarity, and ordinal scales).

## Datasets

- **SemEval-2017 Task 4** — total ?; splits: train (-1), test (-1)

## Metrics

- `macro-average recall` **(primary)** — range: [0, 1]
  - The unweighted mean of recall scores for each sentiment class (Positive, Negative, Neutral). Calculated as the sum of per-class recalls divided by the number of classes.

## Input / output format

**Input**: Tweet text, optionally accompanied by topic labels or user demographic metadata.

**Output**: A sentiment label (Positive, Negative, Neutral) for classification, or a predicted distribution over sentiment classes for quantification tasks.

## Scoring recipe

```python
def macro_avg_recall(y_true, y_pred, classes=['Positive', 'Negative', 'Neutral']):
    recalls = []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        recalls.append(tp / (tp + fn) if (tp + fn) > 0 else 0.0)
    return sum(recalls) / len(recalls)
```

## Common pitfalls

- Confusing macro-average recall with micro-average recall or accuracy, which can yield different rankings.
- Failing to account for class imbalance when comparing against simple baselines (e.g., predicting all Positive or all Negative).
- Mixing up topic-specific sentiment (Subtasks B-D) with overall tweet sentiment (Subtask A), which requires different model architectures.

## Evidence (verbatim from paper)

> For English the best ranking teams were BB_twtr and DataStories, both achieving a macro-average recall of 0.681.

## Citation

```bibtex
@misc{rosenthal2017semeval,
  title={SemEval-2017 Task 4: Sentiment Analysis in Twitter},
  author={Rosenthal et al. (2017)},
  year={2017},
  note={arXiv:1912.00741}
}
```

- arXiv: 1912.00741