Seismic Picker Eval

---
name: seismic-picker-eval
description: Evaluates the performance of deep learning and classical seismic phase pickers across three tasks: event detection, phase identification, and onset time picking. It probes cross-domain transfer capabilities and robustness to varying signal-to-noise ratios and waveform characteristics. Use when the user wants to benchmark on LenDB, GEOFON, INSTANCE, SCEDC, STEAD, ETHZ, Iquique, NEIC, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.13671
  bibtex_key: munchmeyer2021picker
  confidence: high
---

# seismic-picker-eval

> Which picker fits my data? A quantitative evaluation of deep learning based seismic pickers — Münchmeyer et al. (2021) (arXiv:2110.13671, 2021)

## What this evaluates

Evaluates the performance of deep learning and classical seismic phase pickers across three tasks: event detection, phase identification, and onset time picking. It probes cross-domain transfer capabilities and robustness to varying signal-to-noise ratios and waveform characteristics.

## Datasets

- **LenDB** — total ?; splits: test (-1)
- **GEOFON** — total ?; splits: test (-1)
- **INSTANCE** — total ?; splits: test (-1)
- **SCEDC** — total ?; splits: test (-1)
- **STEAD** — total ?; splits: test (-1)
- **ETHZ** — total ?; splits: test (-1)
- **Iquique** — total ?; splits: test (-1)
- **NEIC** — total ?; splits: test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic (ROC) curve. Ranges from 0 to 1, where 1 indicates perfect detection and 0.5 indicates random guessing. It averages performance across all possible decision thresholds.
- `MCC` — range: [-1, 1]
  - Matthews Correlation Coefficient. Measures the quality of binary/multiclass classifications. Symmetric and independent of positive/negative class choice. Ranges from -1 (total disagreement) to 1 (full agreement).
- `MAE` — range: seconds
  - Mean Absolute Error of onset time residuals (predicted pick time minus manual reference time). Measures average magnitude of errors without considering direction.
- `RMSE` — range: seconds
  - Root Mean Squared Error of onset time residuals. Sensitive to outliers, providing a measure of the standard deviation of the prediction errors.

## Input / output format

**Input**: Seismic waveform traces (typically 3-component) provided as fixed-length input windows (e.g., 27 s, 60 s) containing background noise and seismic arrivals.

**Output**: Per trace: detection probability/score, predicted phase label (P, S, or noise), and predicted onset time (sample index or seconds).

## Scoring recipe

```python
# Task 1: Event Detection (AUC)
fpr, tpr, _ = roc_curve(gold_labels, pred_scores)
auc = auc(fpr, tpr)

# Task 2: Phase Identification (MCC)
mcc = matthews_corrcoef(gold_labels, pred_labels)

# Task 3: Onset Time (MAE/RMSE)
residuals = [pred_time - gold_time for pred_time, gold_time in zip(predictions, golds)]
mae = mean(abs(residuals))
rmse = sqrt(mean([r**2 for r in residuals]))
```

## Common pitfalls

- AUC values can be dominated by the asymptotic behavior of ROC curves, potentially masking poor performance in the low false-positive rate regime required for practical deployment.
- Onset time metrics (MAE/RMSE) are highly sensitive to dataset-specific labeling artifacts, such as fixed pick positions within input windows, which can artificially inflate performance for models with large receptive fields.
- Phase identification scores (MCC) can be misleading on datasets with severe class imbalance (e.g., very few S-wave picks), as the model may appear to perform adequately on the majority class while failing on minority phases.

## Evidence (verbatim from paper)

> We evaluate the first task, event detection, using receiver operating characteristics (ROC) and the corresponding area under the curve (AUC). An AUC of 1 indicates a perfect model, an AUC of 0.5 a coin toss. We evaluate task 2 using the Matthews correlation coefficient (MCC). The MCC is symmetric, i.e., in contrast to the AUC or F1 score independent on a choice of positive and negative class. It takes values between -1 (total disagreement) and 1 (full agreement). For evaluating task 3, we use the residuals, i.e., the differences between ML-pick time and the manual reference pick time. We analyze the fraction of samples with high residuals ( >0.45 s for regional, >1.5 s for teleseismic), the root mean squared error (RMSE), and the mean absolute error (MAE).

## Citation

```bibtex
@misc{munchmeyer2021picker,
  title={Which picker fits my data? A quantitative evaluation of deep learning based seismic pickers},
  author={Münchmeyer et al. (2021)},
  year={2021},
  note={arXiv:2110.13671}
}
```

- arXiv: 2110.13671