NewsITA Software has big goals for Needle as it manipulates World Cup historyThis article was originally published onBy Dennis Schaal | June 25, 2010 Some ITA Software engineers have tossed aside airline matters and are focusing on which World Cup host country most recently lost a final (Sweden in 1958) and which nations have mounted two-goal comebacks.Known for its QPX airfare shopping and pricing engine, ITA is developing a new business line, Needlebase. The product, in beta and being tested by around several hundred users, is known in-house as Needle -- as in finding a needle in a haystack.Gwen Beaven, the director of software development for ITA's Needle group, says Needle is a platform for acquiring data from all sorts of disparate sources -- XML feeds, Web scrapes and CSV files, for instance -- merging and editing the data, and publishing the data or exporting them to another database.One of the cool things about Needle is that it converts all of the multi-format data into structured data, enabling users then to write more easily to the data for customization purposes, Beaven says.Another key attribute is that Needle uses machine learning to gather data from the Deep Web, meaning it can retrieve data hidden behind forms that Web crawling usually cannot access, Beaven says.So, to get some bang out of a demo, ITA used Needle to present World Cup History in a way that it's probably never been sliced and diced before.Users can view all the latest stats about the 2010 FIFA World Cup South Africa, but also comb through World Cup history to view the highest scoring matches or the biggest losers, if you're feeling that way yourself.ITA scoured publicly available databases to extract the World Cup data and analyzed every match in World Cup history, Beaven adds.Glenn McDonald, product design and product manager of Needlebase, explains that Needles' World Cup effort "is not much of a cross-source integration effort, by Needle standards, but our idea of 'not much' does still involve combining a fair number of pieces."In fact, about 15 data sources were used and there also was some human intervention for data correction, he adds.For example, McDonald says Needlebase gets all of the basic 2010 match and player information, FIFA rankings and ongoing match results from FIFA.com, but identified many errors and had to fix them using information from places such as Wikipedia, WorldCup-History.com and Planet World Cup."For semantic Web reference, I then added dbpedia links to countries, cups and final matches," McDonald says. "These were scraped from dbpedia in three sets and reconciled with the existing data. And I actually did a couple of more sets of semantic web cross-references that are not shown in the public data because they only make sense for machine consumption."The flag icons came from a publicly available graphics, "but the set [of flag icons] and their country mappings were also Web-scraped and reconciled with the FIFA country list," McDonald says.Needle is an ITA-hosted service and likely will use a subscription model to bring in revenue, although commercialization discussions about the product are believed to be ongoing.Needle is not merely a travel offering.Current beta users are trying it out for job listings, real estate, medical information and conference listings, says ITA spokeswoman Cara Kretz.But, ITA is also showing how an airline might be able to use Needle to display destination activities.Using ITA's Matrix 2 flight search for a Newark-Boston roundtrip, Needle displays the following events and activities below the search results for some US Airways departures, for example.With airlines hungry for ancillary revenue, they might make some money by leveraging companies within the long tail of travel -- like the Concord Band at Fruitlands Museum in Harvard, Mass., or the 25th Annual Putnam County Spelling Bee, playing at Marblehead Little Theater in Marblehead, Mass.Beaven says Needle collects information from around 300 websites, by scraping tour and activity websites where web-scraping is permitted, and through agreements with vendors including OpenTable, Ticketmaster and TicketCity.ITA has around 16 staffers -- including a few interns -- dedicated to Needle and clearly is investing considerable money in the effort.Needle would complement ITA's QPX flight search and still-emerging airline-reservations system businesses.