Phase No Eval

---
name: phase-no-eval
description: Evaluates the ability of neural phase pickers to detect P- and S-wave arrivals in continuous seismic waveforms across multi-station networks. It probes detection accuracy, timing precision, and generalization to out-of-distribution earthquake sequences under varying signal-to-noise conditions. Use when the user wants to benchmark on NCEDC 2020 Test Set, 2019 Ridgecrest Sequence, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.03269
  bibtex_key: sun2023phaseno
  confidence: high
---

# phase-no-eval

> Phase Neural Operator for Multi-Station Picking of Seismic Arrivals — Sun et al. (2023) (arXiv:2305.03269, 2023)

## What this evaluates

Evaluates the ability of neural phase pickers to detect P- and S-wave arrivals in continuous seismic waveforms across multi-station networks. It probes detection accuracy, timing precision, and generalization to out-of-distribution earthquake sequences under varying signal-to-noise conditions.

## Datasets

- **NCEDC 2020 Test Set** — total 43700; splits: test (43700)
- **2019 Ridgecrest Sequence** — total ?; splits: test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, calculated by comparing predicted arrival times against manual ground truth picks.
- `Mean absolute error (MAE) of time residuals` — range: seconds
  - Average absolute difference between predicted and manual arrival times in seconds.
- `Standard deviation of time residuals` — range: seconds
  - Standard deviation of the absolute time differences between predicted and manual picks in seconds.
- `Event match rate` — range: [0, 1]
  - Percentage of ground-truth catalog events successfully matched by predicted events within a 3-second time window.

## Input / output format

**Input**: 30-second (or 60-second for baselines) seismic waveform windows sampled at 100 Hz, with multi-station spatial context provided via graph nodes. Picks are centered in the middle 30 seconds of the window.

**Output**: Probability distribution over time for P- and S-phase arrivals. Final picks are extracted as peaks exceeding a fixed threshold.

## Scoring recipe

```python
picks = [t for t, prob in enumerate(predictions) if prob >= threshold]
tp = len(set(picks) & set(gold))
fp = len(set(picks) - set(gold))
fn = len(set(gold) - set(picks))
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
residuals = [abs(p - g) for p, g in zip(sorted(picks), sorted(gold))]
mae = sum(residuals) / len(residuals)
std = (sum((r - mae)**2 for r in residuals) / len(residuals)) ** 0.5
matched = sum(1 for e_pred in pred_events if any(abs(e_pred - e_gold) <= 3.0 for e_gold in gold_events))
match_rate = matched / len(gold_events)
```

## Common pitfalls

- Models are evaluated at individually optimized F1 thresholds rather than a fixed deployment threshold, which may overstate real-world performance.
- Slightly higher residual standard deviation for PhaseNO is attributed to harder, low-SNR detections rather than model failure, which can mislead readers expecting strictly lower variance.
- Catalog comparisons mix different association algorithms (GaMMA vs. SCSN/template matching) and station counts, making direct event-count comparisons unfair.

## Evidence (verbatim from paper)

> Our method results in the highest F1 scores for both P- and S-waves, being 0.99 and 0.98 respectively. ... PhaseNO results in the smallest mean absolute error for both P and S phases. ... The standard deviation of the pick residuals between SCSN and PhaseNO was 0.10 s for P phases and 0.14 s for S phases ... PhaseNO catalog totaling 26,176 events matched approximately 94% events in the SCSN catalog (10,673 of 11,389) with additional events, indicating the highest recall score of PhaseNO.

## Citation

```bibtex
@misc{sun2023phaseno,
  title={Phase Neural Operator for Multi-Station Picking of Seismic Arrivals},
  author={Sun et al. (2023)},
  year={2023},
  note={arXiv:2305.03269}
}
```

- arXiv: 2305.03269