Mdgender Eval

---
name: mdgender-eval
description: Evaluates models' ability to classify pragmatic gender bias in text across three conversational dimensions: ABOUT (topic-related), AS (speaker role/attitude), and TO (addressee-directed). It probes fine-grained detection of gendered language and contextual bias cues. Use when the user wants to benchmark on MDGENDER, or asks about evaluating this task. Reports percentage accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.00614
  bibtex_key: dinan2020multidimensional
  confidence: high
---

# mdgender-eval

> Multi-Dimensional Gender Bias Classification — Dinan et al. (2020) (arXiv:2005.00614, 2020)

## What this evaluates

Evaluates models' ability to classify pragmatic gender bias in text across three conversational dimensions: ABOUT (topic-related), AS (speaker role/attitude), and TO (addressee-directed). It probes fine-grained detection of gendered language and contextual bias cues.

## Datasets

- **MDGENDER** — total ?; splits: test (-1)

## Metrics

- `percentage accuracy` **(primary)** — range: percent
  - Calculated as the number of correctly classified instances divided by the total number of evaluated instances, multiplied by 100. Only the masculine, feminine, and neutral classes are included; the unknown class is excluded.

## Input / output format

**Input**: Text utterances or sentences annotated for gender bias along the ABOUT, AS, and TO dimensions.

**Output**: A single class label: 'masculine', 'feminine', or 'neutral'.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if gold in ['masculine', 'feminine', 'neutral']:
        total += 1
        if pred == gold:
            correct += 1
accuracy = (correct / total) * 100 if total > 0 else 0
```

## Common pitfalls

- Excluding the 'unknown' class from evaluation, as the paper explicitly states it is not modeled and should be dropped.
- Assuming single-task classifiers generalize across dimensions; the paper shows they specialize to one dimension and fail to transfer.
- Confusing surface-form cues (e.g., pronouns) with contextual bias; ABOUT relies on surface forms while TO/AS require deeper context understanding.

## Evidence (verbatim from paper)

> We measure the percentage accuracy for masculine, feminine, and neutral classes. We do not evaluate on the unknown class, as it is not modeled. Classifier results on MDGENDER are shown in Table 3.

## Citation

```bibtex
@misc{dinan2020multidimensional,
  title={Multi-Dimensional Gender Bias Classification},
  author={Dinan et al. (2020)},
  year={2020},
  note={arXiv:2005.00614}
}
```

- arXiv: 2005.00614