Google On Using Markdown For AI SEO

Google’s John Mueller responded to a submit on Bluesky that lamented all the hassle being wasted on AI agent accessibility as a substitute of specializing in the extra productive exercise of creating websites accessible for people.

Tactical website positioning

One of many constant points of the historical past of website positioning is that practitioners are likely to comply with tactical developments if it’s obvious that everybody else is doing it. I suppose it’s human nature.

Again within the early days there was a man who constructed a listing and bought sufficient excessive PageRank hyperlinks to the house web page to get it to a couple of PageRank of seven (on a scale of 1-10). SEOs had been virtually pushing one another out of the way in which handy this man their cash. He offered hundreds of hyperlinks from his listing, and no one thought to examine if any of the hyperlinks truly made a rating distinction. It was straightforward to examine, however no one did.

The most recent tactic is LLMs.txt and creating markdown pages for AI agent consumption. Cloudflare introduced in February that their infrastructure can routinely create markdown information, largely as a method for builders to avoid wasting LLM token consumption. Their announcement said that OAI-SearchBot, OpenAI’s search crawler, was consuming markdown pages.

OpenAI’s official documentation explicitly positions its OAI-SearchBot as crawling web sites and doesn’t point out or suggest markdown information.

“OAI-SearchBot is for search. OAI-SearchBot is used to floor web sites in search ends in ChatGPT’s search options. Websites which can be opted out of OAI-SearchBot is not going to be proven in ChatGPT search solutions, although can nonetheless seem as navigational hyperlinks. To assist guarantee your website seems in search outcomes, we suggest permitting OAI-SearchBot in your website’s robots.txt file and permitting requests from our revealed IP ranges beneath.”

Sure, it could crawl markdown if it’s pointed to it however that’s not what it’s on the market for.

Stephanie Walter posted:

“Unhappy fact: we’re making the net accessible for AIs, not for folks.
Some websites now provide a textual content model for LLMs, however nonetheless skip actual accessibility wants like correct heading construction, landmarks that display screen reader customers want.”

Google’s John Mueller responded:

“A correctly made web site works properly for AI brokers … and search engines like google and yahoo, and LLMs, and above all, for precise folks.

If you happen to’re attempting to repair accessibility points by making a separate “agent-friendly” model, you’re simply constructing technical debt. You’ll should redo it a number of instances. Simply repair it.”

AI Brokers Crawl HTML

It’s a comparatively trivial factor to crawl HTML. Search engines like google have been doing it for over thirty years. Making websites accessible for everybody is sensible, particularly because it’s consumer conduct alerts which have at all times pushed search engine rankings. From hyperlinks to customers looking with a model title, Google has constantly used consumer alerts for rating functions.

So, from the attitude of website positioning, it makes a load of sense to make web sites accessible for everybody.

Featured Picture by Shutterstock/DETHAL