Regularized Qda Classifier

---
name: tabular-regularized-qda-classifier
description: Use QuadraticDiscriminantAnalysis with regularization for binary classification on data with Gaussian cluster structure
---

# Regularized QDA Classifier

## Overview

Quadratic Discriminant Analysis models each class as a multivariate Gaussian with its own covariance matrix. When features are truly Gaussian-distributed (synthetic data, physical measurements), QDA can outperform tree-based models by directly modeling the decision boundary. The `reg_param` shrinks per-class covariance toward the pooled estimate, preventing singularity on high-dimensional or small-sample data.

## Quick Start

```python
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import roc_auc_score
import numpy as np

oof = np.zeros(len(X))
preds = np.zeros(len(X_test))

skf = StratifiedKFold(n_splits=11, shuffle=True, random_state=42)
for train_idx, val_idx in skf.split(X, y):
    clf = QuadraticDiscriminantAnalysis(reg_param=0.5)
    clf.fit(X[train_idx], y[train_idx])
    oof[val_idx] = clf.predict_proba(X[val_idx])[:, 1]
    preds += clf.predict_proba(X_test)[:, 1] / skf.n_splits

print(f'AUC: {roc_auc_score(y, oof):.4f}')
```

## Workflow

1. Identify that data has Gaussian structure (scatter plots show elliptical clusters)
2. Apply feature selection (VarianceThreshold) to remove noise features
3. Standardize features (optional — QDA is scale-invariant but helps numerically)
4. Train QDA with `reg_param` in [0.1, 0.5] via cross-validation
5. Average fold predictions for test set

## Key Decisions

- **reg_param**: 0.0 = full QDA (each class has own covariance); 1.0 = LDA (shared covariance). Start at 0.5
- **When to use**: data generated by `make_classification`, physical sensor data, or any setting where classes form elliptical clusters
- **Feature count**: QDA estimates O(p^2) parameters per class — reduce features first if p > 50
- **vs LDA**: QDA wins when classes have different covariance shapes; LDA wins when covariances are similar

## References

- [Pseudo Labeling - QDA - [0.969]](https://www.kaggle.com/code/cdeotte/pseudo-labeling-qda-0-969)
- [Instant Gratification (QDA)](https://www.kaggle.com/code/rohandeysarkar/instant-gratification-qda)