Probability of Backtest
Overfitting (PBO): What It Is
and How to Measure It

Probability of Backtest Overfitting (PBO) is the probability that the strategy configuration selected as best on in-sample (IS) data underperforms the median configuration out-of-sample (OOS). It was introduced by Bailey, Borwein, López de Prado, and Zhu in "The Probability of Backtest Overfitting" (2014) as a resampling procedure — Combinatorially Symmetric Cross-Validation, or CSCV — that turns "how likely is it that I just picked the luckiest backtest, not the best strategy?" into a number instead of a hunch.

Why a Single Sharpe Ratio Can't Answer This

Every backtest report shows the performance of the configuration that was kept. Almost none of them show how many configurations were tried and discarded to get there. That omission matters more than it looks: if you backtest 50 variants of a strategy — different lookback windows, different thresholds, different feature combinations — and keep the one with the highest in-sample Sharpe ratio, some of that Sharpe came from genuine signal and some of it came from picking the maximum of 50 noisy draws. Even with zero true edge, the best of 50 random trials will look good by construction. The more configurations you search, the larger that gap gets, and a single point-estimate Sharpe ratio has no way to tell you how much of it is real.

How PBO Is Computed: The CSCV Procedure

The method works on a matrix of returns, not a single equity curve — N strategy configurations (columns) across T time periods (rows). The steps:

1. Split. Divide the T time periods into S contiguous, equal-sized blocks (a common choice is S = 16).
2. Combine. Form every combination of S/2 blocks as the in-sample set, with the remaining S/2 blocks as the out-of-sample set — C(S, S/2) combinations in total.
3. Select. For each combination, rank all N configurations by in-sample Sharpe ratio and note which one ranks first.
4. Check. Find where that same configuration ranks among the N configurations out-of-sample. Convert its OOS rank to a relative rank ω ∈ (0, 1), then to a logit λ = ln(ω / (1 − ω)).
5. Aggregate. PBO is the fraction of combinations where λ ≤ 0 — i.e., where the in-sample winner performed at or below the OOS median.

A PBO above 50% means it's more likely than not that the configuration which looked best in-sample is not the one that will look best out-of-sample — a direct signal that the selection process, not a real edge, produced the top backtest.

PBO vs. CPCV: Related, Not the Same Tool

It's easy to conflate PBO with Combinatorial Purged Cross-Validation (CPCV) because both come out of the same López de Prado methodology and both resample across combinatorial splits. They answer different questions:

They compose naturally: run CPCV on each of your N candidate configurations to get purged, leakage-free OOS return paths for each, then feed those return matrices into the CSCV/PBO procedure to measure whether the config you'd otherwise pick as "best" is a real result or the winner of a search over noise. The CPCV skill file that ships with this site's dataset produces exactly the input PBO needs — a per-path Sharpe distribution instead of one number — for a single strategy; PBO is what you'd run one level up, across the set of strategies that distribution-generation was repeated for.

A Worked Example: Screening Lookback Windows

Take an illustrative case: 50 variants of a Nasdaq-100 cross-sectional momentum strategy, differing only in lookback window (5 to 250 trading days). Backtest all 50 on the same 19 years of point-in-time data, rank by in-sample Sharpe, and the top variant shows Sharpe 1.4. Run the CSCV procedure across all 50 return series and suppose it comes back with PBO = 61%. That result doesn't mean the strategy is worthless — it means that across resampled IS/OOS splits, the configuration that ranked #1 in-sample ranked below the OOS median more often than not. The honest conclusion isn't "discard momentum as a factor," it's "this particular search, over this particular grid, didn't reliably identify a superior lookback window — treat the 1.4 Sharpe as a search artifact, not a finding." A narrower, hypothesis-driven grid (three lookbacks with an explicit economic rationale each, instead of 50 evenly spaced ones) would very likely produce a lower PBO on the same underlying data, because it corresponds to fewer independent draws against the same noise.

Why This Matters More When an Agent Runs the Search

An AI coding agent that can generate and backtest a new strategy variant in seconds removes the natural friction that used to limit how many configurations a researcher would try by hand. That's a genuine productivity gain and also a direct expansion of the search space PBO needs to account for — a hundred agent-proposed variants tested overnight is a hundred independent draws against the same historical noise, whether or not anyone thought of it as "a search" at the time. This site's own autonomous research loop handles this by tracking every experiment — kept and discarded — in an append-only log specifically so the true count of what was tried is never lost; that log is what a real PBO calculation over agent-driven research would need as its sample size. The parallel-agent version goes further and runs a deflated-Sharpe correction — the same multiplicity-correction idea PBO is built on — using the total number of candidates the whole swarm proposed, not just the ones that survived to the final comparison.

Common Questions

What is a good PBO value?

Lower is better, and there's no universally agreed hard cutoff — but well under 50% is a minimum bar (above 50% means the in-sample winner is more likely than not to disappoint out-of-sample), and many practitioners look for PBO under 20% before treating a selected strategy as trustworthy rather than a search artifact.

How is PBO different from p-hacking?

They describe the same underlying failure — trying many things and reporting only the one that worked — but PBO is a measurement of it rather than just a warning against it. P-hacking is the practice; PBO is a specific, computable statistic that quantifies how much a given search process was exposed to it, using the actual returns of every configuration that was tried, not just the one that was kept.

Does PBO replace walk-forward or out-of-sample testing?

No. PBO is a diagnostic on top of a completed search — it tells you whether the selection process itself is trustworthy. It doesn't replace holding out genuine, never-touched out-of-sample data as a final check, and it doesn't replace purged, embargoed cross-validation for evaluating any single candidate's stability. Skipping true out-of-sample validation because PBO looked acceptable is its own way of reintroducing the exact bias PBO exists to catch.

What's the probability of backtest overfitting in practice, for a typical retail strategy search?

There's no single number — it depends entirely on how many configurations were tried and how correlated they are with each other — which is exactly why PBO has to be computed per search rather than assumed. A single hypothesis tested once has a fundamentally different overfitting exposure than a grid search over dozens of parameter combinations, even on identical underlying data.