When right goes wrong for travelers in A/B website testing

A/B testing has become a big part of what online travel agencies do to introduce new functionality and improve the user experience.

But it doesn’t always go to plan.

In a wider discussion on artificial intelligence and machine learning at the Amadeus T3CH event in Madrid this week, Josh Wood, Booking.com's head of global business travel partnerships, shared one such experience.

Wood frames it as an example of “how you can be really right and yet get it really wrong.”

Given the different room types and attributes in a hotel, the idea was to use A/B testing to predict what room business travelers were likely to pick.

It followed a series of experiments where the site gauged the reaction of business travelers according to what was presented to them.

He says: “The calculation to figure out what room they are likely to pick takes a bit of time so we do nothing, then do a calculation without displaying anything but understanding what room you’re likely to pick and then a treatment where we show the actual room.

“The results were conclusive within a day - people hated it with a passion that I never saw with any other experiment I ran at Booking.com, and I ran 250 over a year and a half.”

The next move was to take a step back to see if it was picking the wrong thing but proved that its model was picking the right room 75% of the time.

Wood adds: “Because we said, ‘We think you will like this room,’ not only did people pick anything but that room, they also left the site in droves.

“We talk about personalization ,but it is not this one item; it’s that people really want a curated list.”

During the session, which also included executives from Google Cloud and Air France, panelists were asked whether AI and machine learning are a threat or an opportunity.

Paul Strong, technical director at the office of the CTO for Google Cloud, says it’s about achieving a balance and provides an opportunity to use technology “to serve human beings."

Wood says that at Booking.com it is seen as an “incredible opportunity that is dramatically improving customers’ lives right now.”

He adds that 60% of questions in English coming via the company’s chatbot are now answered using machine learning. The expectation is that the product will continue to evolve over time and soon be available in 10 languages.

Nicolas Nelson, chief information officer at Air France, says the airline sees AI as an opportunity to “really reduce the pain, improve service, improve our cost and improve our quality satisfaction rate.”

* This reporter's attendance at the event was supported by Amadeus.

* Correction: An earlier version of this article quoted Wood as saying the chatbot would answer 100% of queries and 43 languages over the next five years. Booking.com has since clarified its position to say that it is actually evolving over time and will soon be available in 10 languages.