Fibinet Ctr Eval

---
name: fibinet-ctr-eval
description: Evaluates the ability of shallow and deep learning models to predict click-through rates (CTR) on large-scale ad impression datasets. It probes how well architectures can model high-order feature interactions and dynamically weight feature importance using bilinear functions and Squeeze-Excitation networks. Use when the user wants to benchmark on Criteo, Avazu, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1905.09433
  bibtex_key: huang2019fibinet
  confidence: high
---

# fibinet-ctr-eval

> FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction — Huang et al. (2019) (arXiv:1905.09433, 2019)

## What this evaluates

Evaluates the ability of shallow and deep learning models to predict click-through rates (CTR) on large-scale ad impression datasets. It probes how well architectures can model high-order feature interactions and dynamically weight feature importance using bilinear functions and Squeeze-Excitation networks.

## Datasets

- **Criteo** — total 45000000; splits: train (-1), test (-1); repo http://labs.criteo.com/downloads/download-terabyte-click-logs/
- **Avazu** — total 40000000; splits: train (-1), test (-1); repo http://www.kaggle.com/c/avazu-ctr-prediction

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve. Measures the model's ability to distinguish between positive and negative instances across all classification thresholds. Larger values indicate better performance.
- `Log loss` — range: [0, inf)
  - Logarithmic loss measuring the distance between predicted probability distributions and true binary labels. Lower values indicate better calibration and performance.

## Input / output format

**Input**: Sparse categorical and continuous feature vectors representing ad impressions (e.g., 26 categorical + 13 continuous fields for Criteo; 24 fields for Avazu).

**Output**: Predicted probability of a click (binary classification).

## Scoring recipe

```python
def compute_auc(y_true, y_pred):
    return roc_auc_score(y_true, y_pred)
def compute_logloss(y_true, y_pred):
    y_pred = np.clip(y_pred, 1e-15, 1 - 1e-15)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
```

## Common pitfalls

- An improvement of 1‰ (0.001) in AUC is considered practically significant for CTR prediction due to large user bases.
- Log loss is minimized (lower is better), whereas AUC is maximized (higher is better).
- Embedding dimensions and DNN hyperparameters are dataset-specific and require careful tuning to avoid overfitting or optimization difficulties.

## Evidence (verbatim from paper)

> In our experiment, we adopt two metrics: AUC(Area Under ROC) and Log loss. AUC: Area under ROC curve is a widely used metric in evaluating classification problems. Besides, some work validates AUC as a good measurement in CTR prediction*(Graepel et al., [2010](#bib.bib4 ""))*. AUC is insensitive to the classification threshold and the positive ratio. The upper bound of AUC is 1, and the larger the better. Log loss: Log loss is widely used metric in binary classification, measuring the distance between two distributions. The lower bound of log loss is 0, indicating the two distributions perfectly match, and a smaller value indicates better performance.

## Citation

```bibtex
@misc{huang2019fibinet,
  title={FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction},
  author={Huang et al. (2019)},
  year={2019},
  note={arXiv:1905.09433}
}
```

- arXiv: 1905.09433