Does ChatGPT Crawl My Website?

Yes, but "ChatGPT" is three separate crawlers doing three different jobs, not one. Treating them as one is the single most common mistake in a site's robots.txt.

The three OpenAI user agents

• GPTBot: crawls content to train OpenAI's models. This is the one people usually mean when they say "block AI from my site."
• OAI-SearchBot: builds the index that powers ChatGPT's search results. It does not collect content for model training; its only job is search.
• ChatGPT-User: a live, on-demand fetch that happens when a person's ChatGPT conversation follows a specific link, e.g. to answer a question or verify a citation in real time.

Each one respects its own token in robots.txt, independently. OpenAI's own documentation is explicit that you can allow OAI-SearchBot while disallowing GPTBot: the canonical setup for a site that wants ChatGPT search visibility without contributing content to model training.

Why this trips sites up

A common pattern: someone adds a blanket "block all AI bots" rule to robots.txt (often copy-pasted from a template) and it takes out OAI-SearchBot along with GPTBot. The site becomes invisible to ChatGPT search results as a side effect of a training-data decision the owner never actually intended for search.

The opposite mistake also happens: robots.txt looks wide open, but a CDN or WAF rule blocks these user agents at a layer robots.txt can't see, so the crawler gets a 403 that never shows up in a robots.txt audit.

How to check which one is actually happening

• Read your robots.txt literally, token by token. Don't assume a wildcard rule means what you think it means for each specific user agent.
• Check server or edge logs for GPTBot, OAI-SearchBot, and ChatGPT-User requests and their response codes, not just whether requests happened.
• Test from outside your own network. A WAF or bot-management rule can behave differently than a plain robots.txt reading suggests.

Fetchling's scan checks this distinction directly: whether Googlebot, Google-Extended, and OpenAI's crawlers are each actually allowed or blocked, rather than assuming one robots.txt line covers all of them.