Do GPTBot and Other AI Crawlers Support Markdown? Content Negotiation and llms.txt

Published July 28, 2026. The article explains how HTTP content negotiation can serve markdown at the same URL, how llms.txt differs, and whether GPTBot, ClaudeBot, PerplexityBot, and Google-Extended actually request markdown today.

Core answer

• Content negotiation for markdown is a standard HTTP feature.
• A client can send Accept: text/markdown instead of Accept: text/html.
• The server can return markdown and should set Vary: Accept.
• As of the article’s evidence, GPTBot, ClaudeBot, and PerplexityBot do not support markdown negotiation.
• Google-Extended is not a crawler. It is a robots.txt control token.

What content negotiation for markdown is

Content negotiation lets one URL return different representations. A browser usually requests HTML. An agent can request markdown. The underlying URL stays the same.

The article notes that text/markdown was formally registered in RFC 7763 in March 2016.

How the mechanism works

• A request includes an Accept header.
• The server chooses the best available format.
• The response includes the chosen content type.
• Vary: Accept helps caches keep HTML and markdown separate.

The article contrasts this with publishing a separate .md URL or guessing from user agent strings.

Why it matters for GEO

Markdown can reduce payload size by removing navigation, scripts, styling, and other HTML overhead. The article cites Cloudflare’s claim of up to 80 percent smaller payloads and one independent measurement of a 77 percent reduction on a real article.

Clean structure can also make extraction easier for models. The article cites research presented at KDD 2024 that found visibility lifts of up to 40 percent from techniques that improve generative-system parsing and citation.

What the article says about major crawlers

The article also says OpenAI and Anthropic run multiple bots, and that blocking one does not block the others.

llms.txt

llms.txt is described as a discovery file at the site root. It lists important pages before crawling starts. The article says Jeremy Howard proposed it in November 2024.

The article states that Google does not support llms.txt and has no plans to. It also says Google’s June 2026 guidance says no special machine-readable file or markdown is required for Google Search or its generative features.

How to check crawler access

• Read robots.txt for specific bot names.
• Check server logs for real visits.
• Verify crawler IPs against published IP ranges.
• Check firewall and CDN bot-management rules.
• Test pages with JavaScript disabled.
• Use a terminal request such as curl with a crawler user agent.

How to optimize for AI crawlers

• Use semantic HTML with a real heading hierarchy.
• Render essential content in the initial HTML response.
• Add Article or BlogPosting schema where relevant.
• Add FAQPage schema only for genuine FAQ content.
• State answers early and clearly.
• Use content negotiation and llms.txt as additions, not replacements for structure.

Frequently asked questions

Do AI crawlers like GPTBot support content negotiation for markdown?

Not yet, according to the evidence summarized in the article.

Does Google use llms.txt for search?

No, according to the article’s summary of Google’s public statements.

How can I check if AI crawlers can access my content?

Check robots.txt, logs, crawler IPs, firewall rules, and JavaScript-rendered content.

About the author

Satvik Mishra is the Co Founder of Verseodin. The article describes Verseodin as an AI visibility platform that tracks brand citations across ChatGPT, Gemini, Claude, and Perplexity.