Reducing the variance of the average treatment effect estimator is a critical problem in the context of online controlled experiments. Recent developments in variance reduction utilize pre-experiment data to achieve significant variance reduction under the assumption that pre-experiment and in-experiment data are highly correlated. However, in settings such as e-commerce and social media where trends in data may change rapidly, the validity of such an assumption may be questionable. This work addresses this challenge with a two-stage modeling framework that exploits relationships between covariates and the outcome in both pre-experiment and in-experiment data. Inference is made by estimating the counterfactual outcome of each unit and performing a pairwise comparison. This method of inference is proven to be asymptotically unbiased, with an asymptotic variance that scales with the model’s predictive accuracy though is never larger than that of the naive difference in means estimator.