Last week I was invited to speak on a panel at the Boston THack, Tnooz's event for developers and engineers. The topic was innovation in travel technology.
At some point during the discussion at ITA Software's offices, Glenn McDonald, a user interface engineer at Google, who was sitting next to me, commented that "innovation isn’t really a tangible thing".
It’s more a reflection of how we perceive something tangible. Like magic. When we say something is magical, that’s another way of stating that we don’t understand how it works, McDonald surmised.
Real innovation isn't that much different from magic; it occurs when someone solves a problem by removing a constraint that everyone else had taken for granted.
McDonald's analogy inspired me to make two far less elegant and much more incendiary comments, such as “APIs are evil" and "XML is dead; we just don’t know it yet".
This, inevitably, has created a bit of a stir, and I can understand why I need to provide some further explanation.
Of course I don’t actually believe that programming interfaces are intrinsically bad, and I’m sure that XML will be around for decades to come.
What I meant is that the ways in which we store, process and exchange data have become obsolete.
Almost everything in online travel today was engineered under this constraint: data costs money, so we need to be as efficient as possible in handling it.
This has defined how our products and companies have evolved. Data has been locked away in silos. Applications talk to the outside world through throttled peepholes that are only capable of answering specific questions with specific answers.
Big Data is the next frontier. Jim Gray called it the Fourth Paradigm, "analogous to when the printing press was invented".
It has already revolutionised Internet search, social media, astronomy, atmospheric science, genomics, biochemistry, and military surveillance.
In Big Data there are no requests, no predefined parameters and no structured responses. You are free to intersect anything with anything. You can analyse, mutate, group, split, reorder in any way you can imagine.
There are no limits to what you can ask and how often you can ask it. The only constraints are your imagination and the number of computers you have at your disposal.
Suddenly, like magic, anything is possible.
Big Data is not just a set of technologies that can be used to solve hard problems, it is a new way of looking at the problems themselves. And those who master these technologies gain an unfair competitive advantage.
Consider this single example: speed is one of the most important ingredients of a good search experience. In the last fifteen years, no travel company – startup, OTA, airline or GDS – has been able to build and launch an airfare search that returns results in under a second.
One company, a pioneer of Big Data that we all know well, coming from outside of the industry, has just achieved it in less than six months.
We as an industry, and as a community, should consider this our wake-up call. We need to learn the craft of Big Data. We need to invest in the technology of massively parallel processing and scalable storage systems.
Until recently, these technologies were locked away in the IP vaults of big search companies.
But the rise of social media, which has produced more data than had been previously created in all of history of mankind, has forced engineers to re-invent storage, processing, communication protocols – everything.
And Open Source has made these technologies available to any and all of us. Today, they include Hadoop, Disco, Cassandra, HBase, CouchDB, MongoDB and others.
These will evolve over time, but what matters is that learning how to use them will forever change the way we build products and companies.
We also need to take a hard look at all the historical and real-time data that we control, and identify what we can release to the world. We need to create large, open and free data-sets and we need to start publishing real time data streams.
We will be giving up some control and the outcome might appear uncertain. But we cannot not err on the side of excessive caution, because the rewards outweigh the risks. And the cost of doing nothing is greatest of all.
If we do this, others will be inspired to do the same. And this will lead to unpredictable and extraordinary things.
NB: Chime in on Twitter: #bringthemagicback