Same request, completely different hotels
We used a browser in New York and pasted the exact same English request into three separate ChatGPT Temporary Chats. The first and third answers had no hotels in common.
Same dates, budget, neighborhood and language. Each was the first answer in a fresh chat. We never asked it to “try a few others.”
Two answers to the exact same request
The first and third tests are shown, with names and order preserved. The second answer shared Park Terrace with the first and three hotels with the third, so a full swap did not happen every time.
Other AI platforms swapped their New York lists, too
| Platform | Hotel lists | Browser and language | Independent tests compared |
|---|---|---|---|
| ChatGPT | Two completely new lists with no repeated hotel names | New York · English | First vs. third test |
| Claude | Two completely new lists with no repeated hotel names | Taiwan · English | First vs. third test |
| Meta AI | Two completely new lists with no repeated hotel names | New York · Chinese | Second vs. third test |
| Gemini | Some hotel names appeared in both lists | New York · English or Chinese | Completed same-language pairs |
Each test started in a new chat with the exact same request. “First vs. third test” compares the initial answers from two separate chats. Each pair keeps the platform, browser location and language fixed and includes conditional recommendations. Gemini’s Taiwan tests returned only one list per language, so they could not be compared. These examples are too limited to rank platform consistency.
Midtown is a broad area, which may give AI more combinations to choose from. We did not give both cities an equally sized hotel pool or check every property’s availability and total price. The size of the hotel pool is a plausible explanation we still need to test.
Switch languages, and New York’s lists look less alike
Ask ChatGPT for New York hotels in English and Chinese, and the lists have fewer hotels in common. For Taipei, the lists look more alike. We saw that pattern from browsers in both locations.
Hotels recommended in English and Chinese
| Destination | Browser location | English recommendations | Chinese recommendations | Total distinct hotels | Overlap |
|---|---|---|---|---|---|
| New York | New York | 38.5% | |||
| New York | Taiwan | 33.3% | |||
| Taipei | New York | 71.4% | |||
| Taipei | Taiwan | 100% |
“Total distinct hotels” combines both lists, counting shared hotels once. Overlap = hotels in both languages ÷ total distinct hotels.
In the first row, English produced 8 distinct hotels and Chinese produced 10. Five appeared in both languages, giving 13 distinct hotels in total: 5 ÷ 13 = 38.5%.
100% means both languages mentioned the same hotels across the three answers. Order and first choices could still change.
First sightseeing trip to Midtown Manhattan, with a broad neighborhood search.
Business trip near Taipei 101, with a more concentrated location requirement.
The city, trip purpose and search area all changed together, so we cannot pin the difference on the destination or language alone. We have not tested Japanese or other languages yet.
For New York, Claude searched in English either way
We asked Claude for New York hotels from a browser in Taiwan. Every visible web search was in English, whether the request was in English or Chinese.
English requests for Taipei also led to English searches. Ask about Taipei in Chinese, though, and the search language started to vary.
The language we used, and the language Claude searched in
| Destination | Prompt language | Visible web searches |
|---|---|---|
| New York | English | English in all three tests |
| New York | Chinese | English in all three tests |
| Taipei | English | English in all three tests |
| Taipei | Chinese | English, mixed English/Chinese, then Chinese |
English prompts: 19 New York web queries and 8 Taipei web queries, all in English. Chinese prompts: 20 English queries for New York; 2 English and 3 Chinese queries for Taipei. Map searches with no visible query text are excluded.
We can see the words Claude searched for. That does not tell us the language of every page it found or cited, and other platforms may behave differently.
Three search routes, the same Taipei favorite
Claude changed its searches and citations across the Chinese Taipei tests, but kept coming back to Grand Hyatt.
One Chinese request, three visible search paths
English hotel names, room rates and December 2026
KAYAK · Klook · Google Hotels · Trip.com
Hotels near Taipei 101 and nightly prices, in English and Chinese
Agoda · HotelsCombined · trvlogue · yam · UDN
Xinyi district, hotel names and December 2026 prices, in Chinese
Mimi Han · KUO life · V Niu · Xiao Huan Niu
Search topics are summarized in English. We only have the searches and citations visible on screen. We cannot reconstruct every page retrieved or the model’s reasoning from citation order.
The destination does not explain everything: Taipei’s search languages changed with the language we used. We also changed browser-language preferences at the same time, and the two cities had different trip purposes and search areas.
The pages AI kept citing for hotel recommendations
Ask for an $800 total budget in New York, and tax pages enter the conversation. ChatGPT cited hotel FAQs, city tax information and the Hotel Association of New York City’s FAQ.
The pages ChatGPT cited for New York hotels in English
These are all five cited pages saved for this group. Each bar shows how often the page appeared across three answers, counting it once per answer. The bottom four are tied. Frequent citations do not prove a page made AI choose a hotel.
In the third answer, ChatGPT cited the Hyatt FAQ next to a $33 nightly destination fee, and the Arlo FAQ next to a $37 nightly urban fee plus tax. It left both hotels’ budget fit unconfirmed. Those were AI-reported fees from the test; we have not independently verified them as current charges.
Check the charges AI is adding up
A two-night stay in Midtown, with an $800 total budget including taxes and fees
Hotel FAQs and booking pages for mandatory charges, plus the local tax pages AI cites when assessing the total
Suggested check; impact not testedIn Taipei, conference lodging pages made an appearance
In the Taipei tests with Chinese prompts and a browser in Taiwan, ChatGPT cited an ATD conference lodging page in every answer. Its first choices still went Grand Hyatt, Grand Hyatt, then Humble House. The reference stayed; the favorite changed.
The page lists star ratings, reference prices, breakfast, walking distances and booking links in English. It is a useful reminder to check a page’s language as well as where its publisher is based.
A Chinese Claude answer also used reference prices from Mimi Han and KUO life to judge whether Grand Hyatt could fit the $800 budget. Those guide prices were not confirmed totals for the requested dates.
Keep room types, dates, mandatory fees and offer conditions accurate on pages AI cites. Conference hotels should check organizers’ lodging pages, too. We have not established which publisher carries the most weight or whether more coverage leads to more recommendations.
Always on the list, sometimes the favorite
Grand Hyatt Taipei kept making the list in both languages in these selected business-trip tests. English answers sometimes favored Humble House Taipei or Le Méridien Taipei. The Chinese answers kept picking Grand Hyatt.
Which Taipei hotel got the final nod
| Platform & conditions | First | Second | Third |
|---|---|---|---|
| ChatGPTBrowser location: New York Language: English | |||
| ClaudeBrowser location: Taiwan Language: English | |||
| ChatGPTBrowser location: New York Language: Chinese | |||
| ClaudeBrowser location: Taiwan Language: Chinese |
* In the third English ChatGPT answer, a card called Hanns House the “best overall fit,” while the main text suggested looking at Humble House first. The table follows the main text and preserves that inconsistency.
These groups have repeat answers in both languages. Meta often gave several priorities or attached budget conditions, so we show it separately. Gemini left too many answers incomplete for an equivalent comparison.
Meta favored other hotels when budget was a concern
Meta kept mentioning Grand Hyatt, sometimes with a condition: find a discounted rate. By the end of the answer, it often suggested other hotels as safer budget fits.
Where Meta landed in the Chinese Taipei tests
| Browser location | First | Second | Third |
|---|---|---|---|
| New YorkLanguage: Chinese | Le MéridienExplicit preference Grand Hyatt remained a candidate | Grand Hyatt / Le MéridienConflicting signals Grand Hyatt had a preferred label; closing advice listed Le Méridien first | Humble House / SparklePrioritized options Grand Hyatt’s rate needed checking |
| TaiwanLanguage: Chinese | Humble House / SparklePrioritized options Grand Hyatt was near the budget limit | Grand HyattFive-star preference Other candidates were described as safer budget fits | Humble House / EsliteLikely budget fits Grand Hyatt needed an early-booking or member rate |
AI can check the hotel website, then send the traveler to an OTA
In a separate booking study, ChatGPT checked rates on the official hotel website first, but sometimes continued to an online travel agency (OTA). For Regent Taipei, it compared refundable offers, chose the lower-priced Expedia option and reached the guest-details form.
Hotels can track where AI sends travelers to book alongside how often it recommends them. We will share the official-site and OTA comparisons in a separate report.
This observation comes from a separate booking study and is not included in the language experiment’s counts. Testing stopped before payment; no reservation was completed.
So, what should hotels do?
Pick a few requests that important guests would make, then repeat them. One mention is encouraging. The next few answers show whether the hotel keeps its place.
Ask again, and note how the hotel is recommended
Keep the guest, area, budget and language the same. Record main recommendations, conditional options, explicit favorites and reasons for leaving a hotel out. Keep display order separate from the stated preference.
Start with valuable requests where the hotel keeps disappearing
For a Midtown hotel, start with first-time sightseeing trips on a fixed budget. Check the price, location and service claims AI keeps using, then work out which facts need clarification.
Keep a list of the pages AI keeps citing
Group hotel pages, tax information, travel guides, comparison pages and conference lodging pages separately. Check the facts and conditions. If something is wrong, provide an update others can verify.
Follow the languages the target guests use
See which languages and sources turn up for the requests that matter, then decide where translation or local coverage is worth the effort. We have not found a must-use language or must-have publisher for every hotel.
These checks follow from what we saw. We have not tested the effect of content changes or measured any resulting lift in recommendations, bookings or revenue.
Gemini got lost before finding the hotel
Every attempt that failed to produce recommendations in this round was on Gemini. Some returned errors, some kept us waiting, and some answered without recommending a hotel.
The other platforms returned lists, though we still need to check the hotel details and prices.
How often each AI came back with hotels
Gemini returned recommendations in 9 of 12 attempts from the New York browser and 4 of 12 from the Taiwan browser. Differences cannot be attributed to IP location alone or interpreted as a general ranking of model capability.
How we tested, and what we still don’t know
The setup, prompts and calculations are here, along with the limits on what these results can tell us.
The setup and the math
Two browser locations (New York and Taiwan) × two destinations (New York sightseeing and Taipei business travel) × four platforms × two languages × three repeats produced 96 formal attempts. Of these, 85 returned recommendations.
Every request specified December 7–9, 2026, one adult, two nights, a four- or five-star hotel and an $800 total budget including taxes and fees. The prompt did not limit the number of hotels.
We ran English first, then Traditional Chinese, changing the prompt and browser-language preferences together. That means we cannot separate the effects of prompt language, browser language and test order.
Every attempt started in a new chat, using temporary or incognito mode where available. Once AI gave recommendations, we left optional follow-up questions unanswered. We kept errors, timeouts and non-recommendations as outcomes, but left them out of hotel-list comparisons.
To calculate overlap, we combined three answers per language for the same platform, browser location and destination. We divided the hotels shared by both languages by the total distinct hotels across both lists. Main and conditional recommendations count; explicit exclusions and hotels shown only in tool cards do not.
For each cited page, we counted how many answers mentioned it and divided by the attempts in that group. Each page counts once per answer. Search results, image cards, map cards and booking links stay in separate records.
The percentages describe this test, with no claim of statistical significance. They cannot predict how often a hotel will appear for future travelers. The repeat-answer and favorite-hotel comparisons are selected examples.
The AI platforms and browser settings
| Platform | Recorded interface label | Conversation and memory handling |
|---|---|---|
| ChatGPT | GPT-6 / Instant New York records also label it Sol | Temporary Chat |
| Claude | Opus 5.5 / Medium | Incognito; all preference and style settings were not independently verified |
| Gemini | 3.8 Thinking | Temporary Chat; personal-context settings were not all independently verified |
| Meta AI | Instant; underlying model version not displayed | Fresh ordinary conversations; history cleared or initially empty; separate memory state unverified |
Model names follow the labels recorded during testing and may differ from today’s menus. We used consumer chat interfaces in a browser, with no model API calls.
The two locations used different accounts, plans and test times. We checked Taiwan’s location through a network lookup. New York’s location was user-reported, without the same independent IP check.
In the Chinese phase, some product interfaces stayed in English: Gemini in New York, and ChatGPT and Claude in Taiwan. Other available interfaces were switched to Chinese. Location differences therefore cannot be treated as a pure IP effect.
Formal test start times span October 29–30, 2026 UTC. Taipei source-page details received an additional check on November 3, 2026.
Missing records and limits on the conclusions
- We repeated each combination three times. We have not tracked it over days or weeks.
- New York and Taipei differed in destination, search area and trip purpose. The experiment cannot isolate a destination effect.
- The interfaces showed only some searches and sources. We do not have a complete retrieval or crawl log, and citations do not prove what caused a recommendation.
- In six Taiwan-browser Meta conversations, 81 website titles were not preserved. We therefore do not present a complete source-language share or media-influence ranking.
- Some New York-browser ChatGPT citations retained only labels without expanded URLs. Some Gemini sources are also incomplete. The New York citation chart uses the Taiwan-browser records, with preserved domains and paths; URL query strings were not saved.
- Taiwan results came from the delivered transcripts and tables. We have not independently checked every original screenshot.
- The first English Taipei Meta attempt in Taiwan produced A/B answers. Both versions are retained, counting as one attempt.
- Some hotel aliases and branches remain unresolved. The main language comparison is restricted to the selected ChatGPT groups. Home Hotel Xinyi and a branch-unspecified Home Hotel remain separate labels in the Taiwan-browser Taipei set, so its distinct-hotel count is provisional.
- Prices, star ratings, distances and operating status come from AI answers and still need full verification. We did not confirm every total for the requested dates or complete a booking.
Original test prompts
New York sightseeing · English
Help me find a hotel for my first trip to New York. I’m traveling alone, checking in on December 7, 2026 and checking out on December 9. I’d like a four- or five-star hotel in Midtown Manhattan that’s convenient for sightseeing, with a total budget of US$800 for both nights, including taxes and fees.New York sightseeing · Traditional Chinese
幫我找適合的飯店，我第一次去紐約玩，2026 年 12 月 7 日入住、12 月 9 日退房，一個人住
想住曼哈頓中城的四或五星級飯店，方便觀光，兩晚含稅和其他費用的總預算是 800 美元Taipei business trip · English
Help me find a hotel for a business trip to Taipei. I’m traveling alone, checking in on December 7, 2026 and checking out on December 9. I’d like a four- or five-star hotel near Taipei 101, with a total budget of US$800 for both nights, including taxes and fees.Taipei business trip · Traditional Chinese
幫我找適合的飯店，我要去台北出差，2026 年 12 月 7 日入住、12 月 9 日退房，一個人住
想住台北 101 附近的四或五星級飯店，兩晚含稅和其他費用的總預算是 800 美元Appendix: how long a list AI chose to give us
ChatGPT, browser location: New York. In English, each destination produced four hotels in each of the three answers. In Chinese, each destination produced four, five and five hotels.
ChatGPT, browser location: Taiwan. New York produced five, four and four hotels in English, and five, five and five in Chinese. Taipei produced five, four and four in English, and four, five and four in Chinese.
That is how long the lists were in this round. It does not mean Chinese always gets more suggestions, and a longer list is not a quality score.