Sofc Exp Eval

---
name: sofc-exp-eval
description: Evaluates neural models on three information extraction sub-tasks in materials science: detecting experiment-describing sentences, extracting and typing entity mentions (materials, values, devices), and filling experiment-specific slots (e.g., temperature, anode material). Use when the user wants to benchmark on SOFC-Exp Corpus, Synthesis Procedures Dataset, or asks about evaluating this task. Reports macro-average F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2006.03039
  bibtex_key: friedrich2020sofcexp
  confidence: high
---

# sofc-exp-eval

> The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain — Friedrich et al. (2020) (arXiv:2006.03039, 2020)

## What this evaluates

Evaluates neural models on three information extraction sub-tasks in materials science: detecting experiment-describing sentences, extracting and typing entity mentions (materials, values, devices), and filling experiment-specific slots (e.g., temperature, anode material).

## Datasets

- **SOFC-Exp Corpus** — total ?; splits: train (-1), dev (-1), test (-1)
- **Synthesis Procedures Dataset** — total 230; splits: train (200), dev (15), test (15)

## Metrics

- `macro-average F1` **(primary)** — range: [0, 1]
  - Standard F1 score computed as the harmonic mean of precision and recall. Reported as a macro-average across all classes or slots.

## Input / output format

**Input**: Tokenized sentences or documents from materials science publications, processed with ChemDataExtractor for chemical formulas and units.

**Output**: Binary label (experiment vs. non-experiment) for sentence detection; sequence tags (BIO format) for entity mention detection and slot filling.

## Scoring recipe

```python
def compute_macro_f1(predictions, gold):
    precisions, recalls, f1s = [], [], []
    for class_label in unique_classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == class_label and g == class_label)
        fp = sum(1 for p, g in zip(predictions, gold) if p == class_label and g != class_label)
        fn = sum(1 for p, g in zip(predictions, gold) if p != class_label and g == class_label)
        p = tp / (tp + fp) if (tp + fp) > 0 else 0
        r = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
        precisions.append(p)
        recalls.append(r)
        f1s.append(f1)
    return sum(f1s) / len(f1s)
```

## Common pitfalls

- Training sets for sentence detection downsample non-experiment sentences by 0.3, which may inflate test performance if not accounted for.
- Inter-annotator agreement statistics are calculated on a data subset and are explicitly noted as not directly comparable to model scores.
- Entity typing for MATERIAL and DEVICE is significantly harder than VALUE due to rare words and lack of explicit discourse cues.

## Evidence (verbatim from paper)

> We tune our models in a 5-fold cross-validation setting. We also report the mean and standard deviation across those folds as development results. For the test set, we report the macro-average of the scores obtained when applying each of the five models to the test set. Table 7 shows the macro-average F1 scores for our different models on the slot identification task.

## Citation

```bibtex
@misc{friedrich2020sofcexp,
  title={The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain},
  author={Friedrich et al. (2020)},
  year={2020},
  note={arXiv:2006.03039}
}
```

- arXiv: 2006.03039