A/B Test Examples

Dear Growth Designer 👋

I’m Molly, co-leader of the GrowthDesigners.co community.

A/B test examples are THE most popular content in the community and the hardest to come by. Companies don’t like to share their product test results, and designers are often barred by non-disclosure agreements from sharing their work.

So, I lost my sh!t when Mobbin released a new collection of 40+ case studies at ABtest.design. Did we find the treasure trove of product experimentation examples that the community is so hungry for?

Well, yes and no. The collection’s shortcomings include inaccurate case studies, questionable sources, and outdated material. I fact-checked a featured Pinterest case study with a community member who led the work as the senior design manager for growth and he said the actual results were the opposite of what the case study claims.

However, I know how hard it is to find good A/B tests. So proceed with caution, but do take a look!

Your skeptical optimist,

Molly Norris Walker, GrowthDesigners.co Community Leader

-------------------------

About Mobbin

Mobbin is a fully searchable design library of 100,000+ screenshots scraped from the Internet. Unlike Dribbble or Behance, it’s not designer-submitted portfolio work but shipped software. I start every design project by looking for examples on Mobbin. It’s one of only four tools we fully endorse for the community.

Our love of Mobbin is why a lot of us were so excited they were releasing an AB test collection. The AB tests should be used for inspiration-purposes only and shouldn’t be taken as factually accurate.

Proceed with caution. Why? Let’s illustrate with an example.

Pinterest Example

The ABtest.design case study on Pinterest’s logged out experience claims that allowing free content exploration before signup led to a 20% increase in signups. Ok, if you let users view pages without forcing a sign up wall that could be good for increasing signups. But, is it?

I spoke to Pinterest’s Senior Product Design Manager Allen Jordan who is one of our community members. This case study is about his work, but he said they got the outcome and facts wrong. “We actually had less sign-ups initially since we weren't forcing sign-ups anymore,” said Jordan.

So this is the exact opposite of the case study. Sign-ups went down where the case study claims they went up. However, this was still a successful product decision because it positively affected retention.

”We allowed visitors to explore Pinterest more, which led to higher-intent folks signing up. We saw higher quality sign-ups when we removed the sign-up wall to allow for more content browsing, which shot retention through the roof,” said Jordan.

Another win was SEO. Pinterest saw a bump in SEO indexing because search engines can’t crawl behind sign-up walls so more of the logged out product got indexed. The icing on the cake?

“We saw improved user sentiment towards Pinterest since folks would hate tapping Pinterest links from search results and hitting a sign-up wall,” said Jordan.

The Mobbin case study links to a third-party newsletter that doesn’t talk about any of these issues and simply got it wrong.

The Collection’s Shortcomings

• Questionable sources. The Pinterest case study highlights that ABtest.design uses third-party sources such as tear downs, newsletters, and blog posts and not the actual companies or creators themselves.

• Outdated studies. Each case study in ABtest.design has a source link and many of the sources are questionable and/or very old, including a HotelTonight checkout test from 2017. Concepts being tested are things we wouldn’t bother to even question today.

• Not for big bets. The A/B tests in Mobbin’s collection focus on optimizations like adding a pro badge to a paid feature over bigger bets. So it’s not going to be relevant to everyone. For example, I’m a serial head of design turned early-stage venture builder. I founded and grew a zero-to-one startup. None of their examples look at painted door tests or value proposition testing. The collection mostly focuses on single-step enhancements at later-stage scale-ups and companies.

Conclusion

Mobbin is an awesome tool for inspiration. I love that they have done the work for me to sign up and click through real products. I can’t imagine starting a project without looking at Mobbin first. Releasing 40 A/B test case studies is an amazing contribution, but take the test results with a grain of salt. And when in doubt, you can always reach out to GrowthDesigners.co community members in Slack to tell you how it really is. 😉

‍