Making the case for hotels to enable AI 'crawlability'

Leading technology advisors are urging hoteliers to open up to artificial intelligence (AI) search platforms by giving greater access to their crawlers and bots.

Hoteliers should be adding new types of unique content too, as large language models (LLMs) accelerate as a source of direct bookings.

Hospitality companies are in a favorable position in the new “zero-click” environment—where results appear in generative AI summaries on Google or an LLM search such as ChatGPT. A recent Bain study found 80% of consumers rely on zero-click results at least 40% of the time.

While opening up to crawlers can boost visibility and potentially direct bookings, questions remain over security and compliance risks and the longer-term benefits as search-and-book habits evolve.

Access all areas

As Propellic’s Brennen Bliss has previously noted, LLMs are changing how the web is indexed, processed and served to users. His advice? “Serve a crawler-friendly version of your site.”

AI crawlers scour websites and follow links, read publicly accessible content, store information for analysis and generally use what they find to train their LLMs and make sense of the world. It’s different to Google’s Googlebot, for example, which indexes content for search results.

Websites often have a robots.txt file in place, which are publicly available files that tell web crawlers and bots what exactly they are allowed (or not allowed) to access on that site.

Ira Vouk, founder of Hospitality 2.0 Consulting, said larger hotel groups are not open enough to these AI crawlers, leaving franchisee properties underexposed.

That was her conclusion after analyzing several hotel chain websites. In a recent LinkedIn post, she argued these chains were “blocking most AI bots in your robots.txt, which makes you practically invisible on all major AI platforms.”

These files can be viewed by adding robots.txt to the end of a domain name, like https://www.hilton.com/robots.txt or https://www.marriott.com/robots.txt.

Although a rudimentary analysis, it has stimulated debate: Just how far should hotels open up to crawlers?

Vouk believes online travel agencies (OTAs) such as Expedia and Booking.com are more receptive to crawlers and so have an advantage.

“Hotels are losing direct bookings right now because they're not allowing AI bots,” she told PhocusWire. For example, if someone asks ChatGPT about a specific property, Vouk reckons the bot is more likely to source information and a link from the likes of Expedia or Booking.com.

“They're really not showing up in the conversation,” added Pedro Colaco, president and CEO of GuestCentric. “And that is very bizarre, that they're leaving this to third parties.”

PhocusWire contacted Hilton and a number of other hotel groups for comment.

An international spokesperson for Choice Hotels said it is working with its content distribution network provider to allow LLM agents to search its sites.

“We’ve also working with LLM providers to ensure that our sites are visible and searchable on their platforms. We pride ourselves on being industry innovators and embrace technologies that meet travelers wherever they search for hotels to help our franchisees succeed.”

Line of questioning

Another strategy hotels can adopt to grab the attention of crawlers and boost visibility among AI platforms is to incorporate unique information that OTAs are unlikely to feature.

While adding a frequently asked questions (FAQ) section can boost a website’s organic traffic, one expert suggested creating an “IFAQ”—or infrequently asked questions section.

“You should be looking at infrequently asked questions that only you at the hotel can answer,” said Sanjay Vakil, CEO and co-founder at DirectBooker, a PhocusWire Hot 25 Travel Startup for 2026.

“We've typically gone down this (FAQ) list and cut it off and said: These questions are no longer important, they are too long tail,” he said during a recent webinar hosted by direct booking platform Triptease.

“I would encourage people to flip that around. Long-tail queries should be recast as high-intent queries. You can make that information available to them easily within an AI. [Visitors] are going to be more likely to book.”

For example, someone may want to know whether the hotel’s pet policy has a weight limit.

“That's a reasonable question,” Vakil said. Other examples include the temperature of the Jacuzzi.

Handle with care

Granting access to such granular, and sometimes sensitive, data may optimize searchability, but large organizations have their reasons to restrict or reduce access to AI bots.

This can include preventing unauthorized price scrapping, content copying and competitive intelligence gathering, according to Olivier Delaunoy, a hospitality technology expert and former CTO at Bloc Hotels.

“Robots.txt files for groups like Hilton and Marriott are only the visible surface,” he said. “Large hotel brands typically run multiple layers of bot management behind the scenes to block unwanted scraping.”

Meanwhile, disclosing bot-filtering methods in publicly available files effectively helps malicious systems work around them: “A robots.txt file may look simple, but operational security drives the rest,” Delaunoy said.

He added that key players likely have many secondary URL and related microsites, franchise sites, independent hotel pages and older domains still indexed. “These can be discovered by bots even if the main root domain blocks them, leaving them open to AI crawlers,” he said.

However, he admitted visibility can be reduced by blocking AI-driven models, notably because modern AI platforms do not behave like traditional web crawlers.

According to reports, even ChatGPT’s can now pass the “I am not a robot” verification test, and Amazon recently sued Perplexity over its agentic shopping tool, arguing the AI company disguised automated activity as human browsing.

“The best practice is to tailor the hotel bot strategy to block harmful scrapers and allow for beneficial AI crawlers and engage with friendly and approved data-sharing programs, where the hotelier can control accuracy, pricing and brand image,” he added. “Detailed filtering is a skill, and everyone is applying different filters.”

The challenge for hospitality companies is knowing how to detect LLM crawler activity and identify content optimization opportunities. Further ahead, the travel industry needs to work out new types of visibility metrics to assess effectiveness.

Reputation is everything

At the same time, not all businesses are keen to open their doors to AI bots. In July, Internet security provider Cloudflare defaulted to blocking AI crawlers for new domains and introduced a pay-per-crawl model, said Felix Shpilman, CEO of Emerging Travel Group, which owns the RateHawk, ZenHotels and Roundtrip brands.

“This meant property owners, whether large chains or independent hosts, had to manually adjust their settings to allow AI bots access. Many hoteliers are likely unaware of this default setting,” he said.

He also said that while AI platforms can serve as a “promising channel” for direct distribution, they carry a high risk of unauthorized reselling by third parties. “To mitigate this risk, many hotels prefer traditional, controlled channels such as bedbanks and DMCs, which offer clear terms and established agreements,” Shpilman added.

Some media companies have also blocked OpenAI’s GPTBot web crawler from accessing their content.

And will “screen scraping” ever ditch its bad reputation? Only a year ago, a U.S. court ruled Booking.com had violated the Computer Fraud and Abuse Act in a case brought against the OTA  by Ryanair, which claimed Booking.com “screen scraped” its website without permission.

Chase ‘fit and context’

Among all the buzz around AI optimization, GuestCentric’s Colaco offered a word of caution. He feels this new world of search and discoverability is in the phase that SEO was in the beginning, where people were writing keywords in transparent fonts at the end of pages.

“Just like people started to search on Booking.com or on Google, now they start on ChatGPT or Gemini. And it's really a discovery tool, so we should feed it as much information about our hotels as possible,” he said.

As a result, he advises hotels to get back to the basics and tell stories.

“If hotels are telling their story well on their website, everybody that talks about them, whether it's the media or user reviews, if they can surface what the experience is at the hotel, then they'll show up,” he said.

He cited one hotel client, Memmo Alfama in Lisbon, Portugal, which had seen a drop in website visits but an increase in conversion and direct bookings.

“I said they should go back to the original strategy when they launched the hotel, which was about 10 years ago. It was about the neighborhood and its artisans,” Colaco added.

“We're going back to that, where people are going to bring in the overall neighborhood experience into the story and tell that story well. Because people just want to know: What am I going to experience? Travel starts with the discovery piece.”

It’s difficult to find universal agreement on the right strategy regarding AI search, but most will agree that a new chapter is in the process of being written—one that doesn’t rely on traditional search behaviors.

As Barry Diller, chairman and senior executive of IAC and Expedia Group, noted at The Phocuswright Conference in November, “Now that AI has a chance, that search monopoly experience of Google is going to end. What its consequences are to Google, I can't say.”