I was reading Google analytics docs on multi-armed bandit experiments for A/B testing. https://support.google.com/analytics/answer/2844870?hl=en&ref_topic=2844866

In summary, assuming 100 visits/day and 4% conversion rate (CvR) with page A, a scientifically significant experiment for page B to say that the new version actually generates CvR of 5% will require 66 days with the multi-armed approach, or 223 days with the naif 50%-50% approach.

To me, even 66 days seem an enormous amount of time for an early stage startup (say, 1k-to-100k users, 10% active users, growing at 5%/week), where decisions need to be made in a few days. Assuming you can pay for ads, you can probably increase the visits/day, but then the CvR drops down. So, unless you're testing a really bad A page vs a really good B page, I can't figure out an example where the time required is acceptable (say, max 1 week).

I was wonder if anyone has real numbers that also satisfy a more theoretical appetite ;)