Conducting Backtest Validation

---
name: conducting-backtest-validation
description: Structures backtesting methodology with out-of-sample testing, cross-validation, and overfitting detection techniques. Use when validating backtests, detecting overfitting, or ensuring backtest robustness.
tags:
  - process
  - quantitative-capital-strategies
metadata:
  author: casemark
  practice_areas:
    - Quantitative Investing
    - Systematic Strategies
    - Factor Investing
  document_types:
    - Process Documentation
  skill_modes:
    - Process Management
---
# Conducting Backtest Validation

Structures backtesting methodology with out-of-sample testing, cross-validation, and overfitting detection techniques for systematic and factor-based investment strategies.

## When To Use

- Validating a new trading strategy or alpha signal before live deployment
- Auditing an existing backtest for overfitting, look-ahead bias, or survivorship bias
- Comparing multiple strategy variants to select a robust candidate
- Reviewing third-party backtest results (fund managers, vendors, research papers)
- Stress-testing a strategy across regime changes, drawdown periods, or tail events

## Inputs To Gather

- **Strategy specification**: signal logic, universe definition, rebalance frequency, position sizing rules, and transaction cost assumptions
- **Data sources**: price/return series, factor exposures, benchmark indices — confirm whether adjusted for survivorship bias and corporate actions
- **Backtest parameters**: start/end dates, in-sample vs. out-of-sample split dates, walk-forward window lengths
- **Cost model**: commissions, slippage estimates, borrowing costs, market impact assumptions [VERIFY against actual execution data if available]
- **Benchmark(s)**: relevant index or factor portfolio for performance attribution
- **Prior test count**: number of strategy variations already tested on the same dataset (needed for multiple-testing adjustment)

## Workflow

### 1. Data Integrity Audit

- Confirm no look-ahead bias: signals use only information available at decision time
- Check for survivorship bias in the universe (delisted securities, index reconstitution)
- Verify point-in-time correctness for fundamental data (restatements, reporting lags)
- Validate price data for stock splits, dividends, and corporate actions
- Flag any gaps, stale prices, or anomalous returns exceeding ±50% in a single day

### 2. In-Sample / Out-of-Sample Design

- Split the dataset with a minimum 30% out-of-sample holdout; prefer chronological split over random
- Define walk-forward windows: typical choices are 3–5 year in-sample with 1-year forward test steps
- For shorter histories, use k-fold combinatorial purged cross-validation (CPCV) with an embargo period equal to the strategy's maximum holding period to prevent leakage

### 3. Overfitting Detection

- **Multiple-testing adjustment**: apply the Deflated Sharpe Ratio (DSR) or Bonferroni/BHY correction based on the total number of strategy trials
- **Parameter sensitivity**: vary key parameters ±20% and check whether Sharpe ratio degrades more than 30% — flag fragile strategies
- **Minimum Backtest Length (MinBTL)**: estimate required sample size for statistical significance given observed Sharpe; reject if actual history is shorter [VERIFY formula assumptions against strategy frequency]
- **Probability of Backtest Overfitting (PBO)**: run CPCV and compute the share of OOS combinations that underperform the benchmark — PBO > 0.40 is a red flag

### 4. Performance & Risk Decomposition

- Report annualized return, volatility, Sharpe ratio, Sortino ratio, max drawdown, and Calmar ratio for both IS and OOS periods
- Decompose returns via factor attribution (market, size, value, momentum, quality at minimum) to isolate residual alpha
- Examine hit rate, profit factor, and average win/loss ratio at the trade level
- Compute turnover and net-of-cost performance; reject strategies where costs consume >50% of gross alpha

### 5. Regime & Stress Analysis

- Segment performance by market regime: rising rates, falling rates, high vol (VIX > 25), low vol, recession (NBER-dated), expansion
- Identify maximum drawdown duration and recovery period
- Run Monte Carlo reshuffling of trade returns to build confidence intervals around key metrics
- Test sensitivity to execution delay (T+0 vs. T+1 vs. T+2 entry)

### 6. Replication & Documentation

- Record exact signal definitions, universe filters, and rebalance rules so the backtest is fully reproducible
- Log software version, random seeds, and data vendor/snapshot date
- Archive parameter search space and total trial count for future multiple-testing reference

## Output

Produce a **Backtest Validation Report** containing:

1. **Executive summary**: strategy description, headline OOS metrics, and pass/fail recommendation
2. **Data quality findings**: any biases detected, data gaps, or corrections applied
3. **IS vs. OOS comparison table**: side-by-side metrics with statistical significance notes
4. **Overfitting diagnostics**: DSR, PBO score, parameter sensitivity heatmap
5. **Factor attribution**: gross vs. residual alpha, factor loading stability over time
6. **Regime analysis**: performance table segmented by macro regime
7. **Cost impact**: gross vs. net Sharpe, breakeven cost threshold
8. **Recommendation**: deploy, refine, or reject — with specific conditions or thresholds for promotion to paper trading

## Quality Checks

- OOS Sharpe ratio is statistically distinguishable from zero at the 95% level (t-stat > 1.96 after multiple-testing adjustment)
- PBO < 0.40 and DSR remains positive after accounting for all trials
- No single regime drives more than 60% of cumulative OOS profit
- Parameter sensitivity analysis shows smooth, not cliff-edge, degradation
- Transaction cost assumptions are realistic — cross-check slippage with actual fill data or TCA reports [VERIFY against broker/execution platform data]
- Factor exposures are stable and intentional; unintended loadings are flagged
- All data sources and methodology steps are documented sufficiently for independent replication