Seisclip Eval

---
name: seisclip-eval
description: Evaluates a seismology foundation model's ability to classify seismic event types, localize epicenters and depths, and determine focal mechanisms using multi-modal seismic data. It probes cross-dataset generalization and compares fine-tuned, frozen, and scratch-trained variants against spectrum-based baselines. Use when the user wants to benchmark on PNW dataset, SCSN dataset, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.02320
  bibtex_key: si2023seisclip
  confidence: high
---

# seisclip-eval

> SeisCLIP: A seismology foundation model pre-trained by multi-modal data for multi-purpose seismic feature extraction — Xu Si et al. (2023) (arXiv:2309.02320, 2023)

## What this evaluates

Evaluates a seismology foundation model's ability to classify seismic event types, localize epicenters and depths, and determine focal mechanisms using multi-modal seismic data. It probes cross-dataset generalization and compares fine-tuned, frozen, and scratch-trained variants against spectrum-based baselines.

## Datasets

- **PNW dataset** — total ?; splits: test (-1)
- **SCSN dataset** — total ?; splits: test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, computed per class and averaged for macro-average classification performance.
- `MAE` — range: other
  - Mean Absolute Error between predicted and true values for epicenter distance, depth, coordinates, and magnitude.

## Input / output format

**Input**: Multi-modal seismic data including time-frequency spectra and phase/source information; for focal mechanism analysis, multi-station features are input.

**Output**: Classification labels (e.g., earthquake, explosion, surface event, or fault type: normal/reverse/strike-slip) or continuous regression values (epicenter distance, depth, coordinates, magnitude).

## Scoring recipe

```python
def compute_auc(y_true, y_scores, classes):
    auc_scores = []
    for cls in classes:
        y_bin = (y_true == cls)
        auc_scores.append(roc_auc_score(y_bin, y_scores[cls]))
    return sum(auc_scores) / len(auc_scores)

def compute_mae(y_true, y_pred):
    return mean(abs(y_true - y_pred))
```

## Common pitfalls

- STEAD pre-training dataset only contains earthquakes, creating a domain shift that hurts scratch/frozen models on downstream tasks requiring explosions or surface events.
- Focal mechanism analysis is converted from a complex regression problem (strike, dip, rake) to a simpler 3-class classification due to limited training data, which may obscure fine-grained mechanism estimation.

## Evidence (verbatim from paper)

> For all classes, the Area Under Curve (AUC) values of the fine-tune model were consistently better than those of the spectrum-based baseline model. Moreover, across all statistical results, the AUC values of the fine-tune model consistently outperformed the frozen model and the model from scratch.

## Citation

```bibtex
@misc{si2023seisclip,
  title={SeisCLIP: A seismology foundation model pre-trained by multi-modal data for multi-purpose seismic feature extraction},
  author={Xu Si et al. (2023)},
  year={2023},
  note={arXiv:2309.02320}
}
```

- arXiv: 2309.02320