Mobile A/B testing made easy

We do a lot of A/B testing at Socialcam. Web optimization and testing is a mature field, with many great articles (http://37signals.com/svn/posts/2991-behind-the-scenes-ab-testing-part-3-final http://20bits.com/articles/statistical-analysis-and-ab-testing/) These techniques require a little tweaking in a release-software model where new code can't be pushed every day. We've built a fairly simple system that allows us to run mobile A/B test with minimal friction. A few people have emailed about how we do this. I'll describe the entire system here.
 
The first piece is analytics. We use Mixpanel for this. It's a great product. Mixpanel allows us to log events with associated properties, and provides a web interface to slice and analyze the results. Basically, Socialcam is chock-full of mixpanel events. Now, this does get expensive (especially given our recent growth! Ouch Mixpanel bill!). Rather than reduce our logging, however, we decided to sample users. That is, only one out of every N users logs anything to Mixpanel. We get complete data on a sample of our users.     
 
The next component of our testing system is our configuration manager. The configuration manager is a simple class that, on app launch, sends a MAC address (unique device ID) to our server and pulls down a JSON dictionary with key-value configuration pairs. Then, it provides a bunch of accessor methods like:
 
- (id) objectForKey:(NSString *)k default:(id)d;
- (int) intForKey:(NSString *)k default:(int)d;

Each accessor takes a default value, and returns this default if no configuration value is found for the provided key. This allows us to sprinkle server-configurable values throughout the code. Anytime we write code with a value that might ever need to be changed, we wrap it in a configuration manager call, passing in an initial value as the default. This is easy, requiring no work outside of a source file we're editing, but means that we can change the value later by returning a value from the server. In fact, we consider every string in the app changeable, and wrap every one in a macro that maps to a config lookup with the string as the key and default. Thus, any text, button color or cache time can be changed on the fly, without a code push!

So, using Mixpanel we have a detailed record of every user of our app. And with our configuration manager, we have the ability to make major UI changes in a per-device basis without pushing code. All that remains is to track those changes. To do that, there is a special dictionary that is pulled from the configuration manager on app launch. This Experiments dictionary is just a set of key-value pairs that are registered as super properties with Mixpanel (properties that are included in every event). So, if we want to test making the login button red (or giving the video capture view a snappy title; or suggesting more user on account creation) we set config keys on a sample of users to make this change, set a 'button_color':'red' pair in the Experiments dictionary. Then, to evaluate the results, we go into Mixpanel, and slice our existing funnels on the 'button_color' property. No code pushed. The full power of Mixpanel funnels to evaluate the results.

Problem solved!  

Ammon Bartram - CTO Socialcam

3 responses
Hi Ammon,

I found this post through the CocoaControls newsletter—great writeup!

This sorta A/B-testing is exactly what I'd been thinking of doing for an app I'm working on as well, but had been wondering if Apple would be OK with "dynamically-changing" code based on constants passed in from a server..

..it seems like they're cool with it though ;)

Regarding the price of Mixpanel: had you guys considered using something opensource like https://github.com/robbiehanson/CocoaLumberjack instead? I personally haven't used Mixpanel, but incorporating CocoaLumberjack into my project was easy-peasy, and the granularity/control it gives you over logging is pretty awesome. Plus, you can throw in error logging, server-response time logging, hell, anything you can code up, and send it all back to your servers!

I'm guessing though that one of the main selling points of Mixpanel would be the nice UI for slicing and dicing all of that log data? I was thinking of using https://github.com/fpillet/NSLogger (via a bridge like https://github.com/steipete/NSLogger-CocoaLumberjack-connector) to roll my own opensource data analysis solution as well.

Would love to hear your thoughts on ease-of-use/price/constraints of Mixpanel vs. potentially rolling your own solution!

This is a great approach - thanks for sharing. I've created an iOS Client Library partially inspired by this approach and a PHP Server for performing data-driven A/B Split Tests in your iOS Apps - it should give a kick-start to anybody's Mobile A/B testing effort. There are controls (extending UIButton) for A/B testing button text and images as well as the ability to generically split test String values.

Check it out on GitHub.. https://github.com/chrismaddern/iOS-Split-A-B-Test-Library

The iOS split test library provided by Chris looks nice. It's fun that the server code has been commited on github 5 months ago and the client code 11 days ago only :).

We provide the Arise.io service which mainly streamline the deployment of an A/B tested mobile app. It is free for non profit apps. Some guys reading this post might be interested.