I decided to build a tool that lets you input and play with your A/B test data. It should help you understand how the number of users or the conversion rates visually impact the statistical significance.
I actually understood many things while playing with it in a new way, I could even find an interesting way to graphically show the p-value and really SEE what it meant.
Any feedback is appreciated.