Phase Picking Eval

---
name: phase-picking-eval
description: Evaluates a model's ability to detect seismic events and classify phase types (P vs S) from raw waveform windows, particularly under varying amounts of labeled training data. It probes the effectiveness of self-supervised pretraining in learning generalizable seismic features compared to randomly initialized baselines. Use when the user wants to benchmark on ETHZ, GEOFON, STEAD, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.15765
  bibtex_key: liu2024seislm
  confidence: high
---

# phase-picking-eval

> SeisLM: a Foundation Model for Seismic Waveforms — Tianlin Liu et al. (2024) (arXiv:2410.15765, 2024)

## What this evaluates

Evaluates a model's ability to detect seismic events and classify phase types (P vs S) from raw waveform windows, particularly under varying amounts of labeled training data. It probes the effectiveness of self-supervised pretraining in learning generalizable seismic features compared to randomly initialized baselines.

## Datasets

- **ETHZ** — total 22000; splits: train (-1), test (-1)
- **GEOFON** — total 161000; splits: train (-1), test (-1)
- **STEAD** — total 1000000; splits: train (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Computed by plotting the true positive rate against the false positive rate at various threshold levels for the binary event detection task.
- `accuracy` — range: [0, 1]
  - Fraction of correctly classified phase identification samples. Determined by comparing the peak P-phase and S-phase probabilities to assign a class label.

## Input / output format

**Input**: 30-second seismic waveform windows randomly sampled from continuous traces.

**Output**: 3-dimensional probability vector per timestep representing [noise, P-phase, S-phase] probabilities, or class probabilities for classification heads.

## Scoring recipe

```python
def score_event_detection(probs, y_true):
    event_score = 1.0 - probs[:, 0]  # 1 - noise probability
    return roc_auc_score(y_true, event_score)

def score_phase_identification(probs, y_true):
    p_score, s_score = probs[:, 1], probs[:, 2]
    pred = np.where(p_score > s_score, 'P', 'S')
    return accuracy_score(y_true, pred)

def score_onset_regression(probs, y_true):
    onset_time = np.argmax(probs, axis=1)
    return mean_absolute_error(y_true, onset_time)
```

## Common pitfalls

- Using 1 - noise_probability for event detection instead of a dedicated binary classification head.
- Evaluating across varying training data fractions (5% to 100%) rather than a fixed held-out test set.
- Phase identification requires comparing P and S peak probabilities, not taking argmax over the full 3-class vector.

## Evidence (verbatim from paper)

> For a quantitative analysis, we consider the three evaluation tasks defined in the large-scale benchmark by Münchmeyer et al. (2022): 1. Event detection: Given a window of a seismic waveform, determine if it contains an event. 2. Phase identification: Given a window containing exactly one phase arrival, determine if it is a P or an S phase. ... The y axis shows the AUC metric: it represents the area under the curve that plots the true positive rate against the false positive rate at various threshold levels for a binary classification task.

## Citation

```bibtex
@misc{liu2024seislm,
  title={SeisLM: a Foundation Model for Seismic Waveforms},
  author={Tianlin Liu et al. (2024)},
  year={2024},
  note={arXiv:2410.15765}
}
```

- arXiv: 2410.15765