Technology companies, especially consumer facing ones, have seen an explosion in the use of experiments. Many companies now consider statistical hypothesis testing (such as AB testing) to be an appropriate solution to many questions. Outside of a few well-defined examples, however, a lack of statistical knowledge hampers the efforts of real-world practitioners. The purpose of this talk is to document a few experiments and explain why they failed. We will also consider steps that we, as researchers, academics and practitioners, can take to avoid such failures in the future.