NB: This is a guest article by Dorian Harris, founder of hotel booking site Skoosh.
I stumbled across an amazing piece of software recently called Splunk. It does for server logs what Google does for web-pages, more or less.
If you have an in-house geek and large amounts of server data I can't recommend it highly enough. We've uncovered all sorts of strange activity on our site through using it, one of which is the voracious appetite of shopping sites.
These sites lock on to travel websites, whether they are hotels or OTAs, and similar to how Google searches through your code, Splunk runs live searches to access your pricing information.
Unlike the Googlebot, it's a hugely intensive process which our servers can't handle, affecting both Skoosh and its suppliers.
At its worst, it grinds our servers down to a halt and, more regularly, it interferes with the booking process.
As a fierce advocate of transparency on and offline, I find myself strangely uncomfortable with shopping sites.
I don't mind my competitors knowing my prices and I'd be comfortable if there was a central resource where all prices are held. What doesn't work for me is a third party "taking" my pricing information by force and selling it unilaterally.
The obvious solution to this may appear to be to join them. As it happens, it's just impractical.
Just like most OTAs, we don't actually have a database of prices at any one time. We'd like to, but technology and our supplier relationships at this stage just do not allow for it.
For shopping sites to get any meaningful data they have to run tens of thousands of searches on sites per day.
It's the equivalent of a supermarket chain sending in a team of people into the every branch of its rival to note down the prices to the point that the customers can't get to the shelves.
Once I realised the damage these shopping sites were doing at Skoosh, I contacted my competitors and discovered that they've been equally affected.
This begs the question: are this shopping sites just a nuisance, benefiting the few at the expense of others?
Also, what's the legality here? Sure, the pricing information itself is not confidential but how much can you interfere with a competitor's site before it constitutes as DoS Attack?
I was told by the owner of one shopping site that this is "a cost of doing business online", something he equated to similar processes by Google.
Anyway, Google allows you to decide whether or not to spider your site, whereas shopping sites do not. But are they really the same?
It appears that a industry debate is required on the subject.
NB: This is a guest article by Dorian Harris, founder of hotel booking site Skoosh.