<abstract xmlns="http://www.w3.org/1999/xhtml"><p> A standard method to evaluate new features and changes to e.g. Web sites is A/B testing. A common pitfall in performing A/B testing is the habit of looking at a test while it’s running, then stopping early. Due to the implicit multiple testing, the p-value is no longer trustworthy and usually too small. We investigate the claim that Bayesian methods, unlike frequentist tests, are immune to this “peeking” problem. We demonstrate that two regularly used measures, namely posterior probability and value remaining are severely affected by repeated testing. We further show a strong dependence on the prior probability of the parameters of interest.</p></abstract>

A standard method to evaluate new features and changes to e.g. Web sites is A/B testing. A common pitfall in performing A/B testing is the habit of looking at a test while it’s running, then stopping early. Due to the implicit multiple testing, the p-value is no longer trustworthy and usually too small. We investigate the claim that Bayesian methods, unlike frequentist tests, are immune to this “peeking” problem. We demonstrate that two regularly used measures, namely posterior probability and value remaining are severely affected by repeated testing. We further show a strong dependence on the prior probability of the parameters of interest.

Are Multi-Armed Bandits Susceptible to Peeking?

Zagreb International Review of Economics and Business

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

{"article-title":"Are Multi-Armed Bandits Susceptible to Peeking?"}

A standard method to evaluate new features and changes to e.g. Web sites is A/B testing. A common pitfall in performing A/B testing is the habit of looking at...

Are Multi-Armed Bandits Susceptible to Peeking?

Published Online: Jul 03, 2018

Page range: 95 - 104

DOI: https://doi.org/10.2478/zireb-2018-0004

Keywords
multiple comparisons, A/B testing, Bayesian decision theory

© 2018 Markus Loecher, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Are Multi-Armed Bandits Susceptible to Peeking?

Published Online: Jul 03, 2018

Page range: 95 - 104

DOI: https://doi.org/10.2478/zireb-2018-0004

Keywordsmultiple comparisons, A/B testing, Bayesian decision theory

© 2018 Markus Loecher, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Keywords
multiple comparisons, A/B testing, Bayesian decision theory