Mobile A/B testing made easy

We do a lot of A/B testing at Socialcam. Web optimization and testing is a mature field, with many great articles (http://37signals.com/svn/posts/2991-behind-the-scenes-ab-testing-part-3-final http://20bits.com/articles/statistical-analysis-and-ab-testing/) These techniques require a little tweaking in a release-software model where new code can't be pushed every day. We've built a fairly simple system that allows us to run mobile A/B test with minimal friction. A few people have emailed about how we do this. I'll describe the entire system here.
 
The first piece is analytics. We use Mixpanel for this. It's a great product. Mixpanel allows us to log events with associated properties, and provides a web interface to slice and analyze the results. Basically, Socialcam is chock-full of mixpanel events. Now, this does get expensive (especially given our recent growth! Ouch Mixpanel bill!). Rather than reduce our logging, however, we decided to sample users. That is, only one out of every N users logs anything to Mixpanel. We get complete data on a sample of our users.     
 
The next component of our testing system is our configuration manager. The configuration manager is a simple class that, on app launch, sends a MAC address (unique device ID) to our server and pulls down a JSON dictionary with key-value configuration pairs. Then, it provides a bunch of accessor methods like:
 
- (id) objectForKey:(NSString *)k default:(id)d;
- (int) intForKey:(NSString *)k default:(int)d;

Each accessor takes a default value, and returns this default if no configuration value is found for the provided key. This allows us to sprinkle server-configurable values throughout the code. Anytime we write code with a value that might ever need to be changed, we wrap it in a configuration manager call, passing in an initial value as the default. This is easy, requiring no work outside of a source file we're editing, but means that we can change the value later by returning a value from the server. In fact, we consider every string in the app changeable, and wrap every one in a macro that maps to a config lookup with the string as the key and default. Thus, any text, button color or cache time can be changed on the fly, without a code push!

So, using Mixpanel we have a detailed record of every user of our app. And with our configuration manager, we have the ability to make major UI changes in a per-device basis without pushing code. All that remains is to track those changes. To do that, there is a special dictionary that is pulled from the configuration manager on app launch. This Experiments dictionary is just a set of key-value pairs that are registered as super properties with Mixpanel (properties that are included in every event). So, if we want to test making the login button red (or giving the video capture view a snappy title; or suggesting more user on account creation) we set config keys on a sample of users to make this change, set a 'button_color':'red' pair in the Experiments dictionary. Then, to evaluate the results, we go into Mixpanel, and slice our existing funnels on the 'button_color' property. No code pushed. The full power of Mixpanel funnels to evaluate the results.

Problem solved!  

Ammon Bartram - CTO Socialcam