Malnet Tiny Eval

---
name: malnet-tiny-eval
description: Evaluates graph neural networks for Android malware family classification under intra-family and cross-family distribution shifts. It probes how semantic feature enrichment (function metadata and LLM embeddings) and test-time/domain adaptation methods mitigate performance degradation when models encounter unseen malware families. Use when the user wants to benchmark on MalNet-Tiny, MalNet-Tiny-Common, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.06734
  bibtex_key: tran2025mitigating
  confidence: high
---

# malnet-tiny-eval

> Mitigating Distribution Shift in Graph-Based Android Malware Classification via Function Metadata and LLM Embeddings — Ngoc N. Tran et al. (arXiv:2508.06734, 2025)

## What this evaluates

Evaluates graph neural networks for Android malware family classification under intra-family and cross-family distribution shifts. It probes how semantic feature enrichment (function metadata and LLM embeddings) and test-time/domain adaptation methods mitigate performance degradation when models encounter unseen malware families.

## Datasets

- **MalNet-Tiny** — total ?; splits: test (-1)
- **MalNet-Tiny-Common** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - The proportion of correctly predicted malware family labels out of the total number of test samples. Reported as mean accuracy over 3 independent runs with standard deviation.

## Input / output format

**Input**: Function call graphs (FCGs) representing Android applications, optionally augmented with feature collations such as function metadata, LLM-derived code embeddings, and LDP features.

**Output**: A single predicted malware family label per application graph.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Performance is highly sensitive to the feature collation strategy (Trim, Zero, Prune) used to handle missing semantic features in the graphs.
- Results must be averaged over 3 independent runs and reported with standard deviation; single-run evaluations are not comparable to the paper's benchmarks.
- Cross-family (distribution shift) accuracy drops significantly compared to intra-family baselines, so evaluating only on the common split masks the model's generalization capability.

## Evidence (verbatim from paper)

> Tab. 2: Models' accuracy on MalNet-Tiny and MalNet-Tiny-Common across different feature configurations. Top 5 highest values are highlighted in green, darker green represents higher accuracy; values lower than baseline are highlighted in red. Subscript denotes standard deviation over 3 independent runs.

## Citation

```bibtex
@misc{tran2025mitigating,
  title={Mitigating Distribution Shift in Graph-Based Android Malware Classification via Function Metadata and LLM Embeddings},
  author={Ngoc N. Tran et al.},
  year={2025},
  note={arXiv:2508.06734}
}
```

- arXiv: 2508.06734