A few take-aways from a study we ran (~800 consumer queries, repeated over a few days):
* AI answers shift a lot. In classic search a page-1 spot can linger for weeks; in our runs, the AI result set often changed overnight.
* Google’s new “AI Mode” and ChatGPT gave the same top recommendation only ~47 % of the time on identical queries.
* ChatGPT isn’t even consistent with itself. Results differ sharply depending on whether it falls back to live retrieval or sticks to its training data.
* When it does retrieve, ChatGPT leans heavily on publications it has relationships with (NYPost and People.com for product recs) instead of sites like rtings.com
ChatGPT ≠ Google-scale.Google: ~14 B searches/day; ChatGPT: ~37 M (~1 : 400). Only ~15 % of ChatGPT prompts look like classic “search”; most are writing/code tasks. Google’s own search volume grew 22 % in 2024 and still holds >90 % share. An LLM citation is nice for credibility, but it won’t move traffic or revenue anytime soon.
maltelandwehr 2 days ago [-]
The 37M/day is an estimate from Rand Fishkin that often gets quoted as gospel. It is based on limited external data. OpenAI mentioned 1B/day - and had significant overall growth in usage since then.
Also, 1 search on ChatGPT easily replaces 5-10 searches on Google.
Many B2B SaaS companies already get the same amount of leads from ChatGPT that they get from Google. Because clicks from ChatGPT are better informed and have a significantly higher conversion rate. I am talking up to +700% CVR vs traffic from Google for some companies.
edwin 2 days ago [-]
We looked at the same data that Rand Fishkin used and definitely came to a different conclusion.
the_arun 1 days ago [-]
But users are not clicking on search results in google. They get satisfying response from Gemini & end there. It is a good thing for users but bad for inbound traffic to websites.
kibibu 1 days ago [-]
Also bad for users who get completely bullshitted responses.
zdragnar 2 days ago [-]
To add on top of this, how many Google searches also contain Gemini answers in the search results? I've been seeing more and more, especially for code and general factoid searches.
thierrydamiba 2 days ago [-]
Turtles all the way down.
It is definitely interesting to see how much public opinion has been shifting on Google in AI as of late. I wonder what the main force for that is…
echelon 2 days ago [-]
1. From 2010 - 2023(ish), Google slept on DeepMind and allowed OpenAI to steal the narrative. That led to a boom in AI development outside of the Google labs. This is akin to Microsoft's loss of internet/web. A half dozen trillion dollar companies emerged from Microsoft's stumbling, and we're likely to see the same with Google's missteps.
2. After the rise of so many AI startups in the 2020 - 2023 period, the end of Google was being forecast by many. Most of Google's revenue comes through search, and everyone (Nadella, Altman, investors, et al.) were talking about the incremental value of search - Google had to retain 100%, and other players just had to grow.
3. The Google founders, sensing a major blow to their cash cow, took a break from their zeppelin startups and came back. They gave Deep Mind more autonomy, took a knife to product culture, and empowered and encouraged everyone to innovate. The whole company has been re-focused and told they must win AI or face extinction.
4. As of 2025, Google has been killing it on their releases. Gemini, Veo, ... you name it, and they've got industry-leading developments that out-perform and undercut the competition. It's beginning to look like not only will Google not die, but that Google could wipe the field with their AI superiority. It looks like they'll be able to dance circles around OpenAI.
The looming threats are (1) DOJ antitrust eroding Google search ingress and (2) other players stealing Search TAM without the new AI markets being able to replace the search / ads revenue.
Any non-Google players would be wise to put antitrust pressure on Google. Even after the current case ends, they should try to strip away defaults on web and mobile. Stop Google from being able to deploy AI through Google Search, Android, and Chrome. Make Google use the same word of mouth marketing that the rest have to.
It'll be an exciting series of battles ahead.
bravesoul2 2 days ago [-]
Yet.
And even that ignores Google runs an LLM on their search too.
echelon 2 days ago [-]
1. (Anecdotal) I'm barely using Google anymore. I'm using ChatGPT for a ton of queries and getting far better results.
2. Antitrust actions might (should) strip Google of their "panes of glass" with which they force Google Search as the default. Most Google Search queries are simply the result of defaults. Once those defaults are gone, those queries will go elsewhere.
1oooqooq 2 days ago [-]
splitting these are not a good measure. people who know how to search (a skill the latter generations seem to have lost) also searched for the lowest common denominator coding recipes and produced naive code just like people do with current llm models. only sellers of the llm models make the distinction. it's all search.
jameslk 2 days ago [-]
We are in the fleeting era where AI models are not entirely corrupted by marketing and propaganda. Like the early web circa 1990s. It will never be this pure while also being this up-to-date ever again. Enjoy it while it lasts
bloomca 18 hours ago [-]
Yep, I was discussing it with my partner a few months ago. It is just too good right now, in a lot of cases you just slip past all the fluff which are impossible to avoid with traditional search.
The opportunities for AI providers to capitalize on that are too prominent. No idea how long it will take, but imo it's inevitable.
username223 2 days ago [-]
Yep. It won't be long before the web is flooded with pages full of AI-generated content repeatedly mentioning brands near keywords, and "search engines" have been replaced by "summaries" monetized by prompt-stuffing. That's pretty much the extent of these people's genius.
EDIT: I guess the final step is for an "AI agent" to enter your credit card number based on this bot-chat.
AlienRobot 23 hours ago [-]
Yep. Google is an ad company that makes most of its money putting ads on its search page, and Gemini is a Google product that lets you get search results from Google without ads. It doesn't make any sense.
theamk 1 days ago [-]
my "marketing and propaganda", do you mean "other AI model output"?
The internet was pretty bad without AI already, but I can see it heading quickly towards complete nonsense and lack of trust. We are going to have all the same problems we had before AI, but multiplied 100x
1oooqooq 2 days ago [-]
you're literally commenting on a press release by the greediest vc firm. would you care to elaborate your point?
kurtoid 2 days ago [-]
SEO and it's related fields are a net-negative for the Internet (and maybe humanity in general)
xnx 2 days ago [-]
The only good thing about SEO (yuck) is it got some people to care about things that are good for humans too: fast pages, well-structured content, descriptive link text, etc.
bravesoul2 2 days ago [-]
If SEO was literally Search Engine Optimisation it'd be fine.
They really mean SEG (search engine gaming)
godelski 2 days ago [-]
I think the concept of SEO is fine. But the problem comes down to metric hacking.
Certainly you want to make your page easier to index by Google and others, but that's not the only thing that matters. You should improve your content and provide a good product to users. That's what Google intends to measure, but such a thing is actually impossible to do so accurately. So the problem comes down to this stupid cat and mouse game where sites happily shill out links that are immediate turnarounds for users.
I think this is larger than just search. We seem to just be optimizing towards whatever metric we've decided should be used. We then fool ourselves into thinking this proxy actually measures the real thing.
edwin 1 days ago [-]
Unlike classic search, which got worse over time due to SEO gamings, AI search might actually improve with scale. If LLMs are trained on real internet discussions (Reddit, forums, reviews), and your product consistently gets called out as bad, the model will eventually reflect that. The pressure shifts from optimizing content to improving the product itself.
mettamage 1 days ago [-]
I looked into GEO a bit. One of the things I've noticed is that you need to actually optimize for the idea "as if you're talking to a person" and that's because LLMs semantically understand what topics are about. Search engines typically don't, not at that level at least.
fiachamp 1 days ago [-]
Reddit data is already super corrupted with marketers and bot accounts. Why I use https://thegigabrain.com to filter thru the bs on reddit.
gmuslera 2 days ago [-]
I can't imagine how the feedback loop between LLMs optimizing content so it will be picked by LLMs based search engines will end. But it won't be good.
kkaatii 2 days ago [-]
Not too different from how SEO ends. Just spammy content written by bots.
drekipus 2 days ago [-]
there would have to be some added element of human interaction for a a feedback mechanism.
Unfortunately this will be profit driven, rather than something like human enjoyment or insight or something
arnklint 1 days ago [-]
If only you could trust Gemini or gpt to serve truthful answers.
I’m still seeing them concluding the opposite of their own source reference.
duskwuff 1 days ago [-]
Or wildly misinterpreting a source. A few months ago, I saw an especially egregious example where, when asked for the maximum current capacity of a 22 AWG copper wire, Google's AI responded confidently with "551 amps".
The correct answer is two orders of magnitude lower, around 5-7 A. 551 A is the fusing current of that wire - i.e. the current required to make it instantly melt.
meander_water 2 days ago [-]
I came across this recently on Fiverr [0]. I thought it was a joke initially, but the volume of people offering their services implies that there is demand out there somewhere.
It might be that there are more shovel sellers than gold to mine.
nocoder 1 days ago [-]
My experience with the LLM bot is that they are really keen to appease the user and often over confident about their responses. They are prioritizing user engagement over factual nature of the response, it's as if their reward function includes the time spent on the bot. This leads to the bot swaying too much in either direction when it comes to debatable information. They essentially learn what the user prefers and so tend to reinforce those ideas. In some sense, they are like social media influencers who are too confident in their opinion because they are trying to get you to like them. I see us going further into the echo chambers where on the same topic, the bots will give different information to an users based on what the bot thinks about the user preference.
corentin88 1 days ago [-]
> In a world where AI is the front door to commerce and discovery, the question for marketers is: Will the model remember you?
So the question for marketers is: how do you get into the model. And once you are in, how do you outperform others that are in the model too?
corentin88 1 days ago [-]
Maybe content marketing ins’t dead after all. Just the readers have shifted and now you write for LLMs. They will be trained on your articles, to summarize them for LLM readers (aka humans).
sbrother 2 days ago [-]
I launched a product in this space to beta customers just last week -- https://ellm.co -- and the response has been way more positive than I could have hoped for. Every SMB owner I talk to is thinking about this and looking for ways to be ahead of the curve on it even though the number of commercial AI search queries is still dwarfed by Google. It feels like a race, and we are figuring out the rules as we run it.
maltelandwehr 2 days ago [-]
How is ellm different from the more established tools in the market (Profound and Peec AI)?
sbrother 2 days ago [-]
So the short answer is that this space is so early still that there’s plenty of space for more competition. But currently both of those market leaders are targeted at enterprise and have little to no penetration of the SMB/self-serve market. Ellm has a self serve onboarding flow, a $20/month monitoring only tier, and a $100/mo self-serve optimization tier — none of which require scheduling a call with a sales team.
sync 2 days ago [-]
Definitely a bit buggy but looks promising! Try the onboarding flow yourself as a real user (also on mobile!) - particularly leaving the site to research a competitor that is mentioned and then coming back (I got kicked out and had to start from the beginning, at which point it just paywalled me)
It also says I have canceled my subscription at the bottom of the paywall when I never had one. Still, these are little things and I think the bones are theee
sbrother 2 days ago [-]
Thank you so much for giving it a spin and reporting those issues :)
Also if you're interested, here is what your dashboard looks like (the onboarding flow is sadly the buggiest part of the app right now): https://imgur.com/a/eysQzjT
bigbuppo 21 hours ago [-]
And it lost me at "Traditional search was built on links. GEO is built on language."
The fuck it was. How in the hell do they think search works outside the web?
* AI answers shift a lot. In classic search a page-1 spot can linger for weeks; in our runs, the AI result set often changed overnight.
* Google’s new “AI Mode” and ChatGPT gave the same top recommendation only ~47 % of the time on identical queries.
* ChatGPT isn’t even consistent with itself. Results differ sharply depending on whether it falls back to live retrieval or sticks to its training data.
* When it does retrieve, ChatGPT leans heavily on publications it has relationships with (NYPost and People.com for product recs) instead of sites like rtings.com
Writeup: https://amplifying.ai/blog/why-ai-product-recommendations-ke...
Data: https://amplifying.ai/research/consumer-products
Also, 1 search on ChatGPT easily replaces 5-10 searches on Google.
Many B2B SaaS companies already get the same amount of leads from ChatGPT that they get from Google. Because clicks from ChatGPT are better informed and have a significantly higher conversion rate. I am talking up to +700% CVR vs traffic from Google for some companies.
It is definitely interesting to see how much public opinion has been shifting on Google in AI as of late. I wonder what the main force for that is…
2. After the rise of so many AI startups in the 2020 - 2023 period, the end of Google was being forecast by many. Most of Google's revenue comes through search, and everyone (Nadella, Altman, investors, et al.) were talking about the incremental value of search - Google had to retain 100%, and other players just had to grow.
3. The Google founders, sensing a major blow to their cash cow, took a break from their zeppelin startups and came back. They gave Deep Mind more autonomy, took a knife to product culture, and empowered and encouraged everyone to innovate. The whole company has been re-focused and told they must win AI or face extinction.
4. As of 2025, Google has been killing it on their releases. Gemini, Veo, ... you name it, and they've got industry-leading developments that out-perform and undercut the competition. It's beginning to look like not only will Google not die, but that Google could wipe the field with their AI superiority. It looks like they'll be able to dance circles around OpenAI.
The looming threats are (1) DOJ antitrust eroding Google search ingress and (2) other players stealing Search TAM without the new AI markets being able to replace the search / ads revenue.
Any non-Google players would be wise to put antitrust pressure on Google. Even after the current case ends, they should try to strip away defaults on web and mobile. Stop Google from being able to deploy AI through Google Search, Android, and Chrome. Make Google use the same word of mouth marketing that the rest have to.
It'll be an exciting series of battles ahead.
And even that ignores Google runs an LLM on their search too.
2. Antitrust actions might (should) strip Google of their "panes of glass" with which they force Google Search as the default. Most Google Search queries are simply the result of defaults. Once those defaults are gone, those queries will go elsewhere.
The opportunities for AI providers to capitalize on that are too prominent. No idea how long it will take, but imo it's inevitable.
EDIT: I guess the final step is for an "AI agent" to enter your credit card number based on this bot-chat.
The internet was pretty bad without AI already, but I can see it heading quickly towards complete nonsense and lack of trust. We are going to have all the same problems we had before AI, but multiplied 100x
They really mean SEG (search engine gaming)
Certainly you want to make your page easier to index by Google and others, but that's not the only thing that matters. You should improve your content and provide a good product to users. That's what Google intends to measure, but such a thing is actually impossible to do so accurately. So the problem comes down to this stupid cat and mouse game where sites happily shill out links that are immediate turnarounds for users.
I think this is larger than just search. We seem to just be optimizing towards whatever metric we've decided should be used. We then fool ourselves into thinking this proxy actually measures the real thing.
Unfortunately this will be profit driven, rather than something like human enjoyment or insight or something
I’m still seeing them concluding the opposite of their own source reference.
The correct answer is two orders of magnitude lower, around 5-7 A. 551 A is the fusing current of that wire - i.e. the current required to make it instantly melt.
[0] https://www.fiverr.com/categories/online-marketing/generativ...
So the question for marketers is: how do you get into the model. And once you are in, how do you outperform others that are in the model too?
It also says I have canceled my subscription at the bottom of the paywall when I never had one. Still, these are little things and I think the bones are theee
Also if you're interested, here is what your dashboard looks like (the onboarding flow is sadly the buggiest part of the app right now): https://imgur.com/a/eysQzjT
The fuck it was. How in the hell do they think search works outside the web?