Social Chem 101 Eval

---
name: social-chem-101-eval
description: Evaluates a model's ability to reason about and generate rules-of-thumb (RoTs) that capture social and moral norms across 12 distinct dimensions of judgment, such as cultural pressure, legality, and moral foundations. It probes whether neural models can produce attribute-aware, context-sensitive normative judgments for unseen social scenarios. Use when the user wants to benchmark on Social-Chem-101, or asks about evaluating this task. Reports micro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.00620
  bibtex_key: forbes2020socialchemistry101
  confidence: medium
---

# social-chem-101-eval

> Social Chemistry 101: Learning to Reason about Social and Moral Norms — Forbes et al. (2020) (arXiv:2011.00620, 2020)

## What this evaluates

Evaluates a model's ability to reason about and generate rules-of-thumb (RoTs) that capture social and moral norms across 12 distinct dimensions of judgment, such as cultural pressure, legality, and moral foundations. It probes whether neural models can produce attribute-aware, context-sensitive normative judgments for unseen social scenarios.

## Datasets

- **Social-Chem-101** — total 292000; splits: train (-1), val (-1), test (-1)

## Metrics

- `micro-F1` **(primary)** — range: [0, 1]
  - Micro-averaged F1 score computed across all 12 annotation dimensions. True positives, false positives, and false negatives are aggregated globally across all instances and dimensions before calculating precision and recall.

## Input / output format

**Input**: A described social situation or scenario.

**Output**: A set of rules-of-thumb (RoTs) specifying the behavior/action and its acceptability judgment, or a classification label across the 12 social/moral dimensions.

## Scoring recipe

```python
def compute_micro_f1(predictions, golds):
    tp = fp = fn = 0
    for pred, gold in zip(predictions, golds):
        for dim in all_dimensions:
            p = pred.get(dim)
            g = gold.get(dim)
            if p == g == 1: tp += 1
            elif p == 1 and g == 0: fp += 1
            elif p == 0 and g == 1: fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- Performance varies drastically across dimensions (micro-F1 ranges from 0.28 to 0.91), indicating models struggle with nuanced moral/cultural sensitivities.
- The dataset is culturally bounded to English-speaking North American norms, limiting generalizability to other cultures.
- Multiple RoTs can apply to a single situation, requiring models to handle variable-length outputs or multi-label classification rather than single-label prediction.

## Evidence (verbatim from paper)

> The framework demonstrates that neural models can generate attribute-aware, context-sensitive RoTs, though performance varies across dimensions (micro-F1: 0.28–0.91), highlighting gaps in current models' ability to capture nuanced cultural and moral sensitivities.

## Citation

```bibtex
@misc{forbes2020socialchemistry101,
  title={Social Chemistry 101: Learning to Reason about Social and Moral Norms},
  author={Forbes et al. (2020)},
  year={2020},
  note={arXiv:2011.00620}
}
```

- arXiv: 2011.00620