Privacy-Robust Incrementality Estimation in Cookieless Settings via Uplift Modeling: Reproducible Evidence from the Hillstrom E-Mail Experiment

Abstract

Measuring advertising incrementality in the absence of user-level identifiers is increasingly constrained by platform policies and privacy regulations. In cookieless environments, practitioners often observe only aggregated or weak signals (e.g., cohort-level conversion counts) and must still estimate the causal lift of an intervention while quantifying uncertainty. This paper studies cookieless incrementality evaluation through the lens of uplift and individual treatment effect (ITE) modeling under explicit privacy constraints. We conduct full experimental evaluations on the MineThatData (Hillstrom) E-Mail Analytics Challenge dataset (64,000 customers in a randomized controlled experiment with three arms). We cast the task as a binary treatment problem—sending any e-mail campaign versus sending none—and compare six ITE estimators (S-, T-, X-, R-, and doubly robust learners, plus transformed-outcome regression) against cohort-only estimators that emulate cookieless measurement. The cohort estimator uses only aggregated counts and a Bayesian beta–binomial model to shrink noisy rates, and we evaluate robustness under k-anonymity thresholds and Laplace-noised differentially private aggregates. Across held-out test data, the best ID-level model (T-learner with logistic regression) achieves a Qini coefficient of 6.675 and improves the estimated policy conversion rate when targeting the top 20% of customers by predicted uplift. Cohort-only estimation retains a weaker and more variable signal; its point estimate is sensitive to privacy constraints but yields valid uncertainty intervals with 0.892 empirical coverage for a 95% interval in cohort-level validation. The results demonstrate that (i) causal lift is estimable without identifiers when randomized experimentation is available, (ii) doubly robust estimators provide strong performance and fast scoring, and (iii) privacy-preserving aggregation introduces an accuracy–privacy trade-off that can be quantified and monitored using bootstrap and Bayesian uncertainty.