I’m trying to interpret the results of my A/B test for a new website feature, and my confidence interval for the lift in conversion rate includes zero. I know this means the result isn’t statistically significant, but my boss is pushing to roll it out anyway because the point estimate is positive. I’m not sure how to argue against that without a clearer understanding of what the interval actually means for our decision.
