Yesterday Pete Warden, founder and CTO of the social travel photo startup Jetpac, posted on his personal blog an interesting item about the usefulness of analytics for small startups.
Warden, a former software engineer for Apple, notes that it's useful for large companies to run analytics because they have "a big stream of users to run experiments on." Then he asks:
Is it even worth focusing on data when you only have tens of thousands of users?
Warden essentially gives this answer: It depends.
When sample sizes are much smaller, split A/B tests are problematic. Startups need to move quickly, and trying to run experiments that produce statistically meaningful results can be counterproductive, he says.
Yet small startups can benefit from some quick-and-dirty analytics, says Warden, who cites as an example his 18-month-old US startup — which claims "tens of thousands" of users and has scored $2.4M in funding from venture veterans.
In his view, your aim as a startup:
…should be primarily for decision-making ("should we build out feature X?") rather than optimization ("how can we improve feature X?").…
Everything we're looking at should be actionable, should answer a product question we're wrestling with.
Warden created a diagram to illustrate the ideal workflow:
Warden explains the diagram this way:
The silhouettes show where people are looking at the results of our data crunching.
The primary things that everyone on our team religiously watches are the daily report emails, and the UserTesting.com videos that show ordinary people using new features of our app.…
Qualaroo is an awesome offshoot of KissMetrics that we use for in-app surveys, and we also refer to MailChimp's Mandrill dashboard and Urban Airship's statistics to understand how well our emails and push notifications are working.
We have to use App Annie to keep track of our iOS download numbers and reviews over time.
Turning to outside help
Sometimes you can't do it on your own, and startups should hire data experts.
Yesterday, Reuters ran a story on how Warden's startup did this by sponsoring a $5,000 contest on Kaggle, a platform for predictive modelling and analytics competitions.
Within three weeks, the competition's top three teams had more than 85% accuracy in meeting Jetpac's challenge, which was to find the highest quality travel photos that the app's users had posted to Facebook.
Warden told Reuters:
"It made a massive change in terms of the satisfaction the users were reporting."
Read Warden's post, "How do analytics really work at a small startup?" and the Reuters article, "Small business takes on big data."
What has your company's experience been with data analytics?