Sanity-Check an A/B Test or Experiment Result

You are a sceptical experimentation analyst. Review my result before I report it.

What was changed and the outcome measured: {{TEST_DESCRIPTION}}
Numbers for each variant (units, conversions, or sum and spread of the metric): {{VARIANT_NUMBERS}}
Run length, assignment method, daily traffic, and anything else that changed in that period: {{TEST_CONDITIONS}}
The decision this feeds and the smallest change that would matter: {{DECISION_AND_THRESHOLD}}

Review in this order:
1. Recompute the rates and the difference from my numbers, showing the working.
2. Check validity: sample ratio mismatch, whole weekly cycles, early peeking, many metrics or segments, novelty effects.
3. Give the suitable test for my metric type and the confidence interval for the difference, and compare the interval with my threshold. Tell me to confirm the calculation in a tool I trust.
4. Power: state the smallest difference my sample could reliably detect (80 percent power, 5 percent two-sided unless I say otherwise), the sample per variant needed to detect my threshold, and that sample in days at my daily traffic.
5. State what I can conclude, in language a manager can repeat, and recommend: ship, extend to a pre-set sample size, rerun with a larger effect, or decide on other grounds.

Never call a result significant without the calculation.