Cape Wired · GEO & AI Search Guides

Can ChatGPT and AI Search Engines Crawl Your Website?

Can ChatGPT and AI search tools access your website? Learn how crawling works, what blocks bots and which technical issues to check first.

Yes, AI-powered search products can crawl and discover public websites, but the details depend on the platform.

For ChatGPT search, OpenAI uses OAI-SearchBot to discover web content. Google relies on its established Googlebot crawling and indexing systems for Google Search and its AI search features. Perplexity uses PerplexityBot to surface and link web content in Perplexity search.

But allowing a crawler to access your website does not mean the page will automatically be indexed, cited, recommended or shown to a customer.

This guide explains what crawling actually means, how to check whether important pages are accessible, and the technical mistakes that can quietly block search and AI discovery.

What does it mean for an AI search engine to crawl your website?

Crawling is the process of an automated system requesting URLs and reading the content or resources it is allowed to access.

For a traditional search engine, crawling is one stage in a wider process that can include rendering, indexing and later serving the page in response to a search.

Google, for example, describes Search as involving crawling, indexing and serving results. A URL can be crawlable without ultimately being indexed or surfaced for a particular query.

AI search products may combine their own crawlers, search indexes, third-party search providers and other retrieval systems.

You should not assume every platform discovers websites in exactly the same way.

Crawling, indexing, citing and recommending are different things

A technically crawlable page can still fail to appear because it is not indexed, is not considered relevant to the question, provides little useful information, conflicts with another canonical URL, or simply is not selected for the response.

Can ChatGPT crawl your website?

Yes, if your public site allows OAI-SearchBot to access the relevant pages.

OpenAI's publisher guidance says public websites can appear in ChatGPT search and advises publishers who want their content included in summaries and snippets not to block OAI-SearchBot.

OpenAI also says the site host or content delivery network needs to allow traffic from its published IP addresses.

This matters because a robots.txt file can allow OAI-SearchBot while a firewall, CDN or bot-protection service still returns a 403 Forbidden response and prevents the crawler from reaching the page.

Bot mitigation, JavaScript challenges, CAPTCHAs, authentication, geographic restrictions and rate limiting can also prevent automated access.

OAI-SearchBot, GPTBot and ChatGPT-User are not the same thing

This is one of the most important distinctions for businesses reviewing AI crawler settings.

When a user asks ChatGPT or a Custom GPT to visit or use a webpage, OpenAI may make that request using the ChatGPT-User user agent.

Because these actions are initiated by a user, OpenAI says robots.txt rules may not apply in the same way.

Blocking OAI-SearchBot while allowing GPTBot would not be a sensible configuration if your objective is to make the site discoverable in ChatGPT search.

Crawler controls can change over time, so check current platform documentation rather than copying an old robots.txt template from a third-party blog.

Can Google AI crawl your website?

Google's AI Overviews and AI Mode use Google's existing Search infrastructure rather than requiring site owners to optimise for a separate special AI crawler.

Google says the same foundational SEO requirements apply to its AI features.

A page needs to be indexed and eligible to appear in Google Search with a snippet before it can be eligible as a supporting link in AI Overviews or AI Mode.

For Google, the familiar technical questions therefore remain important:

• Can Googlebot access the URL?
• Does the server return a successful response?
• Is crawling blocked by robots.txt?
• Is indexing blocked by a robots meta tag or X-Robots-Tag?
• Can Google render important JavaScript content?
• Can the page be discovered through links or a sitemap?
• Is another URL being treated as the canonical version?

Can Perplexity crawl your website?

Perplexity documents PerplexityBot as the crawler designed to surface and link websites in Perplexity search results.

Its documentation recommends allowing PerplexityBot in robots.txt and permitting requests from its published IP ranges if you want the site to appear in search results.

For PerplexityBot specifically, Perplexity says robots.txt directives are respected.

If a site disallows PerplexityBot, Perplexity says the crawler will not index the full or partial text content of the blocked site, although limited information such as the domain, headline and a brief factual summary may still be indexed.

What is Perplexity-User?

Perplexity also documents a separate Perplexity-User agent for user-triggered page visits.

It is not used for automatic web crawling or foundation-model training. Perplexity says these user-requested fetches generally ignore robots.txt.

What does robots.txt actually do?

A robots.txt file tells compliant crawlers which URLs or paths they are allowed to request.

It is primarily a crawling control.

One common mistake is assuming that blocking a page in robots.txt is the same as removing it from search.

Google explicitly warns that robots.txt is not a reliable method for keeping a web page out of Google Search.

A blocked URL can still sometimes be known from links elsewhere even though Google cannot crawl its page content.

OpenAI similarly notes that if it learns about a disallowed URL through another source, limited link and title information may still be surfaced in some circumstances.

What does noindex do?

A noindex directive tells a search engine that supports the directive not to include the page in its search index.

For Google, this can be supplied through a robots meta tag or an X-Robots-Tag HTTP header.

There is an important technical catch: the crawler needs to be able to access the page in order to see the noindex instruction.

If you block the URL in robots.txt and also place a noindex tag on the page, a crawler may never reach the page to read the noindex instruction.

OpenAI's publisher documentation makes the same practical point for its own crawler controls.

Can a firewall or CDN block AI crawlers even when robots.txt allows them?

Yes.

Modern websites often sit behind Cloudflare, Akamai, hosting security, WAF rules, bot management or rate limiting.

A crawler can be permitted by robots.txt but still fail at the infrastructure layer.

Typical symptoms include:

• 403 Forbidden responses
• 429 Too Many Requests responses
• JavaScript or browser verification challenges
• CAPTCHAs
• country or region blocks
• login requirements
• bot-management rules that reject automated requests
• security plugins that block unknown user agents
• IP allowlists that exclude the crawler

If your infrastructure requires IP-based allowlisting, use the platform's current official IP ranges.

Where practical, combine the expected user agent with official provider IP verification and review server or security logs.

This is why checking robots.txt alone is not a complete crawlability audit.

Can JavaScript stop search and AI systems from understanding a page?

JavaScript can create crawlability and rendering problems, particularly when important content is only added after scripts run or after a user interacts with the page.

Google renders JavaScript using its Web Rendering Service, but it also documents limitations and advises developers to ensure important content can be rendered reliably.

Google specifically warns that lazy-loaded content should not depend on user actions such as clicking or scrolling because Google Search does not interact with the page in the same way as a human visitor.

Other crawlers may handle JavaScript differently.

It is therefore safer to ensure that core business information exists in robust, accessible HTML rather than depending entirely on complicated client-side behaviour.

For a GEO-focused site, that includes:

• the business name and proposition
• product or service names
• descriptions
• prices where appropriate
• locations and service areas
• important specifications
• FAQ answers
• authors or experts
• internal links to relevant pages

Can content hidden behind clicks, tabs or forms be a problem?

Potentially.

A collapsible FAQ that already exists in the page HTML is very different from information that is fetched only after a user submits a form, accepts a browser challenge or performs a complex interaction.

The safest principle is not "never use tabs or accordions".

Does a sitemap make your website crawlable?

A sitemap can help search engines discover URLs, particularly new or updated pages, but it does not override crawling or indexing restrictions.

Putting a blocked, noindex or inaccessible page in a sitemap does not make that page eligible for search.

Think of a sitemap as a discovery aid, not a permission system.

Do internal links matter for crawlability?

Yes. Internal links are an important discovery route.

Google recommends making links crawlable and says important pages should be reachable through links from other findable pages.

A page that exists only in a sitemap, has no navigation route and receives no contextual internal links is harder for users to discover and gives search systems fewer normal paths through the site.

This is one reason the Cape Wired GEO cluster is architected as a connected system rather than a collection of isolated posts.

What HTTP response should an important page return?

For a normal public page that you want crawled, the server should usually return a successful 2xx HTTP response.

Persistent 4xx or 5xx errors, redirect loops and access-denied responses can prevent crawlers from reliably retrieving the content.

A page that loads perfectly in your own browser while logged in is not proof that an external crawler receives the same response.

How can you check whether your website is crawlable?

A practical crawlability check should examine more than one layer.

For larger sites, technical crawling tools can help identify widespread noindex directives, broken links, redirect chains, orphan pages and status-code problems.

A simple robots.txt example for AI search discovery

The exact file should reflect your own site architecture, and you should not paste crawler rules blindly.

Conceptually, a site that wants to allow these major search-discovery crawlers could contain:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

You might make a separate decision about GPTBot depending on your preferences around potential foundation-model training.

ChatGPT-User and Perplexity-User are user-triggered agents rather than the automatic search crawlers shown in this example, so they should be evaluated separately.

Should every page be crawlable?

No.

A good technical strategy is not "let every bot crawl every URL".

Some pages should remain private, excluded from search or protected from unnecessary crawling.

Examples can include:

• customer account pages
• checkout and basket states
• private dashboards
• internal search-result pages
• staging environments
• duplicate parameter URLs
• confidential documents
• thin or utility pages with no search value

The objective is to make pages that explain the business and serve customers discoverable while deliberately controlling pages that should not be public search destinations.

What pages matter most for AI search crawlability?

Prioritise pages containing information a customer or search system would need to understand the business.

• homepage
• About page
• core product or service pages
• category or collection pages
• location pages
• high-value guides
• case studies
• contact and business-information pages
• policy or technical pages supporting important claims where relevant

If these pages are blocked, noindexed, inaccessible behind a firewall or dependent on broken rendering, more advanced GEO work should wait.

What if ChatGPT links to a page that you blocked from crawling?

Blocking crawling does not always mean the URL itself becomes completely unknown.

OpenAI says that if it obtains the URL of a disallowed page from another source and has signals that it is relevant, limited link and page-title information may still be surfaced in some circumstances.

Google similarly explains that a URL blocked by robots.txt can sometimes appear in search results if Google discovers the URL from links elsewhere, even though Google cannot crawl its page content.

If your objective is to prevent a public URL from appearing in search, use the platform-supported indexing controls rather than assuming robots.txt alone is sufficient.

Can you block AI training but still appear in AI search?

For OpenAI, its current controls allow that distinction.

OpenAI uses OAI-SearchBot for ChatGPT search controls and GPTBot for potential foundation-model training controls.

ChatGPT-User is separate again: it supports certain user-triggered visits and is not used to determine whether content appears in Search.

That makes the phrase "AI crawling" unhelpfully broad.

Businesses should ask which crawler they are controlling and for what purpose:

• search discovery
• user-initiated access
• model training
• advertising validation
• other platform-specific functions

Do not assume one allow or disallow rule controls every use of a platform.

How long after fixing crawlability will AI search change?

There is no single cross-platform timetable.

OpenAI says changes to OAI-SearchBot robots.txt settings can take around 24 hours to be reflected in its search systems.

Perplexity says its crawler-setting changes can also take up to 24 hours to propagate.

Google's recrawling guidance is broader: crawling a changed page can take from a few days to a few weeks, and requesting a recrawl does not guarantee immediate inclusion in search results.

After fixing access, verify that the crawler can actually reach the page, then re-check visibility using the same measurement method rather than expecting an immediate AI-answer update.

Does being crawlable make your website easier for AI to understand?

Crawlability gives a system access to the material. It does not guarantee that the material is clear.

A crawler may successfully request a page that still contains vague headings, thin product information, contradictory facts or important details hidden inside images.

That is why crawlability is only the first layer.

The crawlability mistakes we would fix first

These are technical access problems.

Fixing them can remove barriers, but it should not be described as a guarantee that an AI platform will cite or recommend the business.

Where should your business start?

If you are unsure whether AI search systems can access the important parts of your site, start with a technical check before producing large amounts of new GEO content.

Ask:

• Can the relevant crawlers request the page?
• Does the page return a successful response?
• Is the page indexable where indexing is required?
• Can the important content be rendered?
• Can the page be discovered through internal links?
• Are security systems accidentally blocking crawlers?
• Does the canonical configuration point to the correct version?

If the answer to any of those is no, solve the access problem first.