The lookahead bug that makes every backtest look brilliant

Most first backtests are wrong in the same way. The signal is calculated from today's close, and then the trade is filled at today's close. In real trading you cannot know the close until the day is over, so the backtest is buying at a price it only learned about after the moment it acted.

Here is the version with the bug:

import pandas as pd

df["sma"] = df["close"].rolling(50).mean()
df["signal"] = (df["close"] > df["sma"]).astype(int)
df["returns"] = df["close"].pct_change() * df["signal"]

The signal on row t uses the close on row t, and then earns the return of row t. The fix is one line, a shift by one bar:

df["returns"] = df["close"].pct_change() * df["signal"].shift(1)

Now the position you hold today was decided with yesterday's information, which is the only information you had.

How much does it matter

On a 50 day moving average crossover over ten years of daily SPY data, the unshifted version has produced numbers that beat the shifted version by a wide margin in every test I have run. The gap is not a small edge, it is the entire result. Run both on your own data and compare the equity curves before you trust either one.

The general rule

Any column that feeds a decision must only contain information available at the moment of the decision. That covers more than prices. Fundamentals get restated. Index membership is revised. Earnings dates move. When you join a second dataset onto your prices, ask what time that data actually arrived, not what date it is stamped with.

A useful habit is to write the backtest so the position column is always the output of shift. If you never index a decision to the same bar it acts on, this class of bug cannot happen.