Drsm Certified Robustness Eval

---
name: drsm-certified-robustness-eval
description: Evaluates the standard classification accuracy and certified robustness of a malware detector against adversarial byte perturbations. It measures how well the model maintains correct predictions under a bounded perturbation budget using a de-randomized smoothing defense with window ablation. Use when the user wants to benchmark on PACE, or asks about evaluating this task. Reports Standard Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.13372
  bibtex_key: saha2023drsm
  confidence: high
---

# drsm-certified-robustness-eval

> DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified Robustness — Saha et al. (2023) (arXiv:2303.13372, 2023)

## What this evaluates

Evaluates the standard classification accuracy and certified robustness of a malware detector against adversarial byte perturbations. It measures how well the model maintains correct predictions under a bounded perturbation budget using a de-randomized smoothing defense with window ablation.

## Datasets

- **PACE** — total ?; splits: train (-1), validation (-1), test (-1); repo https://github.com/ShoumikSaha/DRSM

## Metrics

- `Standard Accuracy` **(primary)** — range: percent
  - Percentage of files correctly classified. A file is correctly classified if the winning class from majority voting across n ablated windows matches the true label. Ties are broken in favor of 'malware'.
- `Certified Accuracy` — range: percent
  - Percentage of files where the certified robustness condition holds: n_m(x) >= n_b(x) + 2Δ, where n_m and n_b are votes for malware and benign, and Δ is the perturbation budget. Represents a theoretical lower bound on accuracy under adversarial perturbation.

## Input / output format

**Input**: Raw binary executable files, processed into n overlapping ablated windows/sequences for classification.

**Output**: Binary classification label (benign or malware) derived from majority voting across n ablated windows.

## Scoring recipe

```python
def compute_metrics(votes_malware, votes_benign, labels, delta):
    # votes_malware, votes_benign: lists of length n (number of windows)
    # labels: ground truth binary labels
    n_files = len(labels)
    
    # Standard prediction: malware if n_m > n_b, else benign (ties -> malware)
    preds = ['malware' if m >= b else 'benign' for m, b in zip(votes_malware, votes_benign)]
    standard_acc = sum(1 for p, l in zip(preds, labels) if p == l) / n_files * 100
    
    # Certified accuracy: check robustness condition for budget delta
    certified_correct = sum(1 for m, b in zip(votes_malware, votes_benign) if m >= b + 2 * delta)
    certified_acc = certified_correct / n_files * 100
    
    return standard_acc, certified_acc
```

## Common pitfalls

- Tie-breaking differs between evaluations: standard accuracy breaks ties in favor of 'malware', while certified accuracy uses a stricter threshold (n_m >= n_b + 2Δ) to maintain consistency with the robustness bound.
- The perturbation budget Δ starts at 2, not 1, because perturbations smaller than the window size can overlap at most 2 ablated sequences.
- Certified accuracy reports a theoretical lower bound; empirical robustness against actual adversarial attacks is typically higher.

## Evidence (verbatim from paper)

> Besides standard accuracy, we also evaluate the certified accuracy for DRSM-n models. Recall that – ‘certified accuracy’ is the percentage of files for which the inequality [1] holds true for DRSM-n models. In short, it denotes the lower bound of model performance even when the attacker can perturb bytes in Δ number of ablated windows and alter predictions for all of them.

## Citation

```bibtex
@misc{saha2023drsm,
  title={DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified Robustness},
  author={Saha et al. (2023)},
  year={2023},
  note={arXiv:2303.13372}
}
```

- arXiv: 2303.13372