September 3, 2026
What OAI-SearchBot actually does, how it differs from GPTBot and OpenAI's other crawlers, how OpenAI discovers and indexes content for ChatGPT search, and exactly what changes, and does not change, when a site chooses to block it.
Most sites that vanish from ChatGPT search never get an explanation for it. Nothing looks broken. Google rankings hold steady, the site loads fine, and whoever handles SEO checked robots.txt months ago and moved on. The actual cause, more often than site owners expect, is a single crawler most people have never heard of: OAI-SearchBot.
GPTBot gets most of the attention, mainly because it arrived first and became shorthand for opting out of AI training. OAI-SearchBot does something else entirely. It is the crawler that decides whether a page can be surfaced, cited, and read inside a ChatGPT search answer, and it runs on its own separate line in robots.txt regardless of whatever a site has already decided about GPTBot. Treating the two as interchangeable, or assuming one setting covers both, is the fastest way for a site to opt out of ChatGPT search without ever meaning to.
This guide covers what OAI-SearchBot actually is, exactly how it differs from GPTBot, how OpenAI's crawlers discover and index the content ChatGPT search draws from, why getting the robots.txt configuration right matters more than it looks like it should, and precisely what does, and does not, change once a site chooses to block it.
OAI-SearchBot is the crawler OpenAI built specifically to power ChatGPT search. It has one job: fetch public web pages so ChatGPT can surface and cite them inside search answers. It is not the crawler OpenAI uses to collect training data, and it is not the bot that visits a page because a person pasted a link into a chat. OAI-SearchBot exists for exactly one purpose, making a page eligible to appear, with a citation, inside a ChatGPT search result.
OpenAI's own documentation puts it plainly: OAI-SearchBot is used to surface websites in search results in ChatGPT's search features, and a site opted out of it will not be shown in ChatGPT search answers. That single sentence is also why OAI-SearchBot needs its own explanation separate from GPTBot. Confusing the two, or assuming one robots.txt line covers both, is how a site quietly disappears from ChatGPT search while believing it only opted out of AI training.
As an OpenAI web crawler, OAI-SearchBot identifies itself through a user agent string that always contains the text OAI-SearchBot, close to this: compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot (the version number changes over time). OpenAI also publishes the exact IP ranges this crawler operates from at openai.com/searchbot.json, so a site owner can confirm a given visit is genuine rather than a spoofed user agent claiming to be OAI-SearchBot.
Search OAI-SearchBot vs GPTBot and the short answer is this: GPTBot collects content that may train OpenAI's models, OAI-SearchBot collects content that can appear in ChatGPT search answers, and robots.txt treats them as two completely separate permissions. Allowing one says nothing about the other.
Purpose: GPTBot feeds model training. OAI-SearchBot feeds ChatGPT's search feature.
Effect of blocking: disallowing GPTBot means nothing crawled going forward can be used in a future round of model training. Disallowing OAI-SearchBot means the site stops being eligible for ChatGPT search answers specifically.
Independence: each setting works on its own. A site can allow OAI-SearchBot for search visibility while disallowing GPTBot for training, or the reverse, with no crossover between the two.
Shared crawling: OpenAI has noted that if a site allows both bots, it may reuse the results of a single crawl for both purposes instead of crawling the same page twice, though the robots.txt permissions themselves stay independent.
GPTBot and OAI-SearchBot are not the only ChatGPT crawler agents either. OpenAI documents two more: ChatGPT-User, which visits a page only when a person directly asks ChatGPT to look at something, and OAI-AdsBot, a narrower crawler that checks whether an ad's destination page meets ChatGPT's safety and policy standards before that ad can run. Neither is used for training, and neither is controlled through the same robots.txt line as OAI-SearchBot. The next section covers specifically how OAI-SearchBot builds what ChatGPT search actually has available to cite.
OAI-SearchBot discovers content the way most modern search crawlers do: it follows links outward from pages it already knows about, requests each new URL, and adds what it finds to the pool of content ChatGPT search can read from and cite. This is the ChatGPT Search crawler in the literal sense, a program that reads robots.txt, respects what it says, then fetches and refetches pages on an ongoing basis so ChatGPT's search feature has something current to draw on.
Discovery itself works largely through links. OAI-SearchBot finds new pages mainly by following them, whether from other sites, from a sitemap, or from pages inside a site it has already crawled. A page with no inbound links and no sitemap entry can sit on an otherwise fully open site and simply never get found, which is worth ruling out before assuming a robots.txt problem is the reason a page never shows up in ChatGPT search. Keeping an XML sitemap current and linking new content from pages already crawled both help OAI-SearchBot find a page in the first place, well before robots.txt or content quality ever come into play.
OAI-SearchBot's systematic crawling is not the only way ChatGPT reaches a page, either. ChatGPT-User complements it by fetching one specific page in real time whenever someone typing into ChatGPT asks it to read that page right then, entirely separate from the background crawl OAI-SearchBot runs on its own schedule. how ChatGPT actually sources everything that ends up in an answer covers where that real time fetch fits alongside training data and uploaded files as a third input, for the fuller picture beyond crawling alone.
Getting OAI-SearchBot robots.txt configuration wrong is one of the more expensive, least visible mistakes a site can make, because nothing about it looks broken. The site still loads. Google still ranks it. The only symptom is that ChatGPT search quietly never mentions it, and there is no error message telling anyone why.
The mistake usually happens the same way. A site added a rule disallowing GPTBot at some point, reasonably enough, to keep its content out of AI training. If that same rule, or a broader wildcard rule aimed at unrecognized bots, also catches OAI-SearchBot, the site has opted out of ChatGPT search answers without ever meaning to. Each of OpenAI's crawlers reads its own line in robots.txt, so a rule written for GPTBot does not automatically apply to OAI-SearchBot, but a wildcard rule, or a rule that lists bots by an outdated or incomplete name, can sweep it in by accident.
This is worth checking now rather than after noticing a drop in ChatGPT referral traffic, since the two are hard to connect after the fact. A quick read of a site's live robots.txt file, matched against the exact user agent tokens OAI-SearchBot and GPTBot, takes a few minutes and answers a question that otherwise stays invisible until someone happens to ask why a well regarded brand never seems to come up in a ChatGPT search answer. It is one piece of the wider technical GEO work of making a site genuinely readable to AI systems , not a standalone checkbox.
Recommended Tool
Don't guess whether ChatGPT, Claude, Gemini, or Perplexity can access your content. Analyze your site in seconds.
Check AI Visibility Try Prompt Finder
No signup required • Instant results
Configuring, optimizing, and monitoring access breaks down into a handful of ongoing habits rather than a single setup step, and the same three habits apply whether the crawler in question is OAI-SearchBot, GPTBot, or any other OpenAI crawler documented in a site's robots.txt file.
A minimal working example for separating the two decisions looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
This example allows OAI-SearchBot everywhere while keeping GPTBot out, a common choice for a site that wants ChatGPT search visibility without opting into model training. Reverse the Allow and Disallow lines for the opposite decision, or set both to Allow if training use is acceptable too.
Verify against published IP ranges, not the user agent string alone. A user agent string is text any script can fake. OpenAI publishes its real OAI-SearchBot IP ranges at openai.com/searchbot.json, and matching server log hits against that list is the only reliable way to confirm a visit claiming to be OAI-SearchBot actually came from OpenAI.
Check firewalls and CDN bot rules separately from robots.txt. Robots.txt only states an intention; whether a request actually reaches the server is decided by a different system entirely, and security tooling set up long before OAI-SearchBot existed can quietly reject it no matter what the file says. a full walkthrough of checking whether AI crawlers can actually reach your content goes through this line by line.
Give OAI-SearchBot something worth citing once it arrives. Access alone does not guarantee a citation. Content that states a direct answer early, uses a real heading hierarchy, and loads inside the page's initial HTML response gives OAI-SearchBot something usable the moment it does arrive.
Monitor whether access is converting into visibility, not just whether it exists. Confirming OAI-SearchBot can reach a page is a one time check. Knowing whether that access is actually showing up as citations and mentions inside ChatGPT search is an ongoing question, closer to tracking brand visibility and citations across AI search than a robots.txt audit.
One distinction worth keeping straight while doing all of this: robots.txt controls access, full stop. A file like llms.txt is a different, optional mechanism for pointing agents toward a site's most useful pages once they are already allowed in. what llms.txt actually does and how to build one covers that separate piece in depth, but no llms.txt file makes up for getting the OAI-SearchBot and GPTBot robots.txt lines wrong.
Blocking OAI-SearchBot removes a site from ChatGPT's synthesized search answers. OpenAI is direct about the mechanism: once a site opts out through OAI-SearchBot, it stops being one of the sources ChatGPT search reads and cites when building an answer. A raw link to the site can still turn up elsewhere in ChatGPT's output, but the site will not be surfaced as a cited source inside a synthesized search answer itself.
Competitors that still allow OAI-SearchBot become the available material instead. A brand that would otherwise be a natural answer to a question like best options for a given product category simply stops being something ChatGPT search has to work with, no matter how strong that brand's actual reputation or product might be. The absence is total from ChatGPT search's point of view, not partial, since there is nothing halfway about a source it was never allowed to read in the first place.
This takes effect on a delay, not instantly. OpenAI has noted it can take roughly 24 hours from a robots.txt change for its systems to catch up, so a block added today will not necessarily be reflected in ChatGPT search behavior within the hour, and neither will a change that reopens access after a block is lifted.
Blocking OAI-SearchBot is a narrower action than it sounds, and knowing its limits matters as much as knowing its effect.
AI Visibility Live preview
Check whether AI engines can reach, read and recommend your pages.
It does not stop GPTBot from crawling for training. These are separate permissions. Blocking OAI-SearchBot alone leaves GPTBot exactly as allowed or disallowed as it already was.
It does not stop ChatGPT-User. A person who directly pastes a link into ChatGPT can still trigger a live fetch of that exact page through ChatGPT-User, since that agent runs on its own permission, and a request triggered by a person asking directly does not get filtered through robots.txt the way an automated crawl does.
It does not guarantee removal from every ChatGPT surface. As covered above, the site can still surface as a plain, uncited reference elsewhere in ChatGPT's output, even once OAI-SearchBot is blocked.
It does not affect other platforms' crawlers. Blocking OAI-SearchBot has no bearing on Googlebot, Google-Extended, ClaudeBot, Claude-SearchBot, or PerplexityBot. Each needs its own separate robots.txt line for a site that wants the same opt out elsewhere.
It does not retroactively remove content already used. If OAI-SearchBot indexed a page before a block was added, the block stops future crawling, not anything already incorporated from earlier visits.
It does not affect standard search rankings. OAI-SearchBot is specific to ChatGPT search. Disallowing it has no bearing on how a site performs in Google, Bing, or any other traditional search engine.
No. OAI-SearchBot's documented purpose is surfacing pages inside ChatGPT search, not collecting material for training. GPTBot handles training separately, and the two operate under independent robots.txt permissions, so allowing OAI-SearchBot is not the same as opting into training use.
Match the visit against the IP ranges OpenAI publishes at openai.com/searchbot.json rather than trusting the user agent string alone, since any script can claim to be OAI-SearchBot in its request headers. A log entry with the right user agent text but an IP address outside that published range is worth treating as suspicious.
No. ChatGPT-User runs on its own permission, separate from OAI-SearchBot, and fires only in response to someone typing a request inside ChatGPT itself rather than through ongoing automatic crawling. A site can turn off OAI-SearchBot's search access entirely while ChatGPT-User still fetches that same site whenever a user explicitly requests it.
OpenAI's own guidance points to roughly 24 hours for a robots.txt update to be reflected in how OAI-SearchBot behaves, whether the change opens access or closes it. A site should not expect a same day reversal in ChatGPT search visibility right after editing the file.
No. OpenAI runs a separate agent, OAI-AdsBot, whose only job is confirming a submitted ad's destination page is safe and policy compliant before that ad runs. It plays no role in search visibility or training, and it answers to its own robots.txt line, independent of OAI-SearchBot.
What Is OAI-SearchBot?
Differences Between GPTBot and OAI-SearchBot
How OpenAI Crawls and Discovers Content for ChatGPT Search
Why Your Robots.txt Configuration for OAI-SearchBot and GPTBot Matters
OAI-SearchBot Best Practices: How to Configure, Optimize, and Monitor Crawler Access
What Happens When You Block OAI-SearchBot?
What Blocking OAI-SearchBot Does Not Do
Frequently Asked Questions
Summarize with ChatGPT Summarize with Claude Summarize with Perplexity
S
Satvik Mishra
Co Founder of Verseodin
Satvik Mishra is the Co Founder of Verseodin, an AI visibility platform that tracks how brands are cited across ChatGPT, Gemini, Claude, and Perplexity. He writes about generative engine optimization, technical AEO, and what it actually takes for a brand to show up inside AI generated answers.
Newsletter
Get the latest AI Visibility insights, GEO research, product updates, and SEO strategies delivered straight to your inbox.
Continue improving your AI visibility.
Discover what users are asking AI about your industry and uncover prompt opportunities.
Open Prompt Finder
Analyze whether AI search engines can discover and cite your website.
Run Free Report
AI SEO 10 min read
A step by step framework for finding missing keywords, separating page level gaps from domain level gaps, and turning competitor research into a prioritized content plan you can actually execute.
Read article
AI SEO 11 min read
AI brand mentions explained: what they are, how ChatGPT, Gemini, and Perplexity decide which brands to name, the factors behind them, and the metrics that show whether you are winning.
Read article
AI SEO 11 min read
The five signals AI systems use to understand, validate, and retrieve brands, and how trust, authority, and data consistency decide which brands get surfaced in AI answers.
Read article
Want more AI SEO insights?
Monthly research on AI search, GEO and citation trends. No noise.
[ partner program ]
Join our partner network, whether you're an agency, consultant, or reseller. Leave your email and we'll reach out with details.
Revenue share
Earn a cut of every referral you bring on, renewing month over month.
Co-marketing
Joint webinars, case studies, and content with the Verseodin team.
Priority support
Direct line to our team plus early access to new features.
What Is OAI-SearchBot? How OpenAI Crawls and Discovers Content for ChatGPT Search | VerseOdin