Counterfactual Detection Eval

---
name: counterfactual-detection-eval
description: Evaluates a model's ability to detect counterfactual statements in product reviews. It probes robustness to selection bias from clue phrases, cross-lingual transfer via machine translation, and the effectiveness of different sentence encoders and classifiers on imbalanced binary classification tasks. Use when the user wants to benchmark on Multilingual Counterfactual Detection Dataset (Amazon Reviews), or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.06893
  bibtex_key: oneill2021counterfactual
  confidence: high
---

# counterfactual-detection-eval

> I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviews — O'Neill et al. (2021) (arXiv:2104.06893, 2021)

## What this evaluates

Evaluates a model's ability to detect counterfactual statements in product reviews. It probes robustness to selection bias from clue phrases, cross-lingual transfer via machine translation, and the effectiveness of different sentence encoders and classifiers on imbalanced binary classification tasks.

## Datasets

- **Multilingual Counterfactual Detection Dataset (Amazon Reviews)** — total 24000; splits: train (-1), test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall. Standard binary classification metric.
- `Matthew's Correlation Coefficient (MCC)` — range: [-1, 1]
  - Correlation coefficient between observed and predicted binary classifications. Accounts for class imbalance and incorporates all correlations within the confusion matrix.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified instances out of total instances.
- `Precision` — range: [0, 1]
  - Proportion of true positive predictions among all positive predictions.
- `Recall` — range: [0, 1]
  - Proportion of true positive predictions among all actual positive instances.

## Input / output format

**Input**: Single sentence text string.

**Output**: Binary label indicating whether the sentence expresses a counterfactual (positive) or not (negative).

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    tp = sum((t == 1 and p == 1) for t, p in zip(y_true, y_pred))
    fp = sum((t == 0 and p == 1) for t, p in zip(y_true, y_pred))
    fn = sum((t == 1 and p == 0) for t, p in zip(y_true, y_pred))
    tn = sum((t == 0 and p == 0) for t, p in zip(y_true, y_pred))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    acc = (tp + tn) / (tp + fp + fn + tn)
    denom = ((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)) ** 0.5
    mcc = (tp * tn - fp * fn) / denom if denom > 0 else 0
    return {'F1': f1, 'MCC': mcc, 'Accuracy': acc, 'Precision': prec, 'Recall': rec}
```

## Common pitfalls

- Accuracy can be highly misleading on this dataset due to severe class imbalance (underrepresented positive class).
- Models may memorize clue phrases used for data selection rather than learning semantic counterfactual cues, leading to overfitting when clues are masked or absent in test data.
- Japanese tokenization (e.g., MeCab) can split or remove verb/adjective inflections that contain counterfactual clues, artificially lowering performance for transformer models using BPE.

## Evidence (verbatim from paper)

> F1, Matthew's Correlation Coefficient (MCC; Boughorbel et al., 2017), and accuracy are used as evaluation metrics. MCC (∈ [-1,1]) accounts for class imbalance and incorporates all correlations within the confusion matrix (Chicco and Jurman, 2020). Accuracy may be misleading in highly imbalanced datasets because a simple classification of all instances to the majority class has a high accuracy. However, for consistency with prior work, we report all three evaluation metrics in this paper. All the reported results are averaged over at least 3 independently trained models initialised with the same hyperparameter values.

## Citation

```bibtex
@misc{oneill2021counterfactual,
  title={I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviews},
  author={O'Neill et al. (2021)},
  year={2021},
  note={arXiv:2104.06893}
}
```

- arXiv: 2104.06893