Phreshphish Eval

---
name: phreshphish-eval
description: Evaluates phishing website detection models on temporally disjoint, real-world data with realistic base rates, while mitigating training-to-test leakage and varying difficulty levels. Use when the user wants to benchmark on PhreshPhish, or asks about evaluating this task. Reports Precision-Recall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.10854
  bibtex_key: dalton2025phreshphish
  confidence: high
---

# phreshphish-eval

> PhreshPhish: A Real-World, High-Quality, Large-Scale Phishing Website Dataset and Benchmark — Dalton et al. (2025) (arXiv:2507.10854, 2025)

## What this evaluates

Evaluates phishing website detection models on temporally disjoint, real-world data with realistic base rates, while mitigating training-to-test leakage and varying difficulty levels.

## Datasets

- **PhreshPhish** — total ?; splits: train (-1), test (-1), benchmark (-1)

## Metrics

- `Precision-Recall` **(primary)** — range: [0, 1]
  - Area under the Precision-Recall curve computed across classification thresholds. Precision is the fraction of predicted phishing sites that are actually phishing, and Recall is the fraction of actual phishing sites correctly identified.
- `Accuracy` — range: [0, 1]
  - Fraction of correctly classified instances (phishing and benign) out of the total dataset size.

## Input / output format

**Input**: URL string and/or full HTML content of the webpage.

**Output**: Binary classification label (phishing/benign) or classification probability.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred_proba, threshold=0.5):
    y_pred = (y_pred_proba >= threshold).astype(int)
    accuracy = np.mean(y_true == y_pred)
    precision, recall, _ = precision_recall_curve(y_true, y_pred_proba)
    pr_auc = auc(recall, precision)
    return {'accuracy': accuracy, 'PR-AUC': pr_auc}
```

## Common pitfalls

- Using publicly available datasets with unrealistically high phishing base rates (e.g., ~33%), which artificially inflate model performance.
- Failing to apply temporal partitioning or leakage filtering, allowing models to memorize similar train/test points and overestimate generalization.
- Ignoring HTML similarity when pruning datasets, as dissimilar URLs can still host nearly identical phishing pages.

## Evidence (verbatim from paper)

> Precision-recall curves on the benchmark datasets. Right: Precision-recall curves on benchmark datasets with and without applying the difficulty filter

## Citation

```bibtex
@misc{dalton2025phreshphish,
  title={PhreshPhish: A Real-World, High-Quality, Large-Scale Phishing Website Dataset and Benchmark},
  author={Dalton et al. (2025)},
  year={2025},
  note={arXiv:2507.10854}
}
```

- arXiv: 2507.10854