# Are the AI crawlers you allow really getting into your site?

> Allowing an AI crawler in robots.txt does not open the door. Our server logs show fake crawlers, crawlers cut off by the host's firewall, and an OpenAI crawler that stops and then comes back. How to check, and what to allow.

- Original page: https://dotsland.com/en/are-ai-crawlers-really-getting-in/
- Language: en
- Published: 2026-11-07T09:00:00+01:00
- Updated: 2026-10-19T17:00:47+02:00
- Section: Digital Marketing
- Publisher: Dotsland, https://dotsland.com/en/

---

## In brief

- Allowing a crawler in robots.txt does not open the door to it. Between the rule and the page, a CDN or a firewall, yours or your host’s, can turn it away, sometimes without a trace.
- A visitor calling itself “GPTBot” is only GPTBot if its IP address sits in the ranges OpenAI publishes. The other major AI companies publish the means to check theirs too.
- On dotsland.com, 130 fake GPTBots went looking for key and secrets files, while the Apple and Amazon crawlers we invite were being cut off by our host’s firewall. They still were on 13 October.
- Measured on 13 October, over 45 days: GPTBot is back, with 466 verified visits and the latest on 11 October, ClaudeBot has 592 verified visits, and every one of our articles published more than a week earlier had been read by at least one verified AI indexing crawler.
- Our own check got Anthropic wrong: it flagged visits from the real ClaudeBot as spoofed, because it did not check against the list the company publishes.
- The check fits in a log export: filter by crawler, check the address, count the response codes, look at the dates.

AI crawlers you allow in robots.txt don’t necessarily get in. Our robots.txt allows twenty-one AI agents by name, and our server logs from 31 July to 12 September 2026 show that some never got in, that others came in under a borrowed name, and that the OpenAI crawler reading our new articles in August stopped afterwards.

## Six weeks of logs contradicted our robots.txt

On 12 September 2026 we pulled the access logs for dotsland.com, 111,113 lines in all. robots.txt open, sitemap up to date, llms.txt file in place: the question looked like a formality. We compared the address of each visit with the ranges published by the crawler’s company, and for Google used reverse DNS, which tells you which domain an address belongs to.

A crawler’s name takes one line to write into any script, and some visitors use it to get in in its place. Of 902 visits claiming to be GPTBot, 772 really came from OpenAI’s addresses. The other 130 left no doubt about their intentions: they asked for files such as `/secrets.json` or `/id_rsa`, where access keys are kept. On the Google side, 197 of the 579 visits announcing themselves as Googlebot came from addresses that did not resolve to Google, about one in three. And fifteen requests claiming to be Perplexity were trying to reach an internal server address, a well-known attack pattern.

> **Our check got it wrong, and that is what makes the method useful**  
> On 12 September, our script flagged as spoofed 385 visits claiming to be ClaudeBot, which came on 13 August from a single address hosted at Amazon. Having found no list from Anthropic, we had checked the address in the network registry, whois. The list does exist, referenced in Anthropic’s help centre, and the address in question sits in one of its ranges: in all likelihood, that was the real ClaudeBot. We made the same mistake with Perplexity, which we thought published nothing and which publishes two address files. You check against the company’s list, reread on the day of the check, never against your memory.

Another trap, the most flattering one: the busiest agent in our logs was Claude-User, sent at the request of a Claude user. A large share of those visits came from our own working sessions, so they measured no audience at all.

The dates said more than the totals. GPTBot read our new articles on 7, 12 and 17 August, one to five days after publication. Then nothing: by 12 September, none of the twelve articles published since had been fetched, even though the crawler had still visited on 4 September. In all, 767 verified visits in August, 5 from 1 to 12 September. OAI-SearchBot, OpenAI’s search crawler, came every day, but 53 of its 76 September visits were for robots.txt. Over a different window, from 31 August to 24 September, we counted only four reads of our llms.txt by a verified OpenAI crawler, as we reported in [what llms.txt v2 changes](https://dotsland.com/en/llms-txt-v2-what-changes/).

> **Update of 13 October 2026**  
> We replayed the check on the last 45 days of logs, from 29 August to 13 October, 92,543 lines in all. GPTBot had stopped in September: it is back, with 466 verified visits and a latest visit on 11 October. OAI-SearchBot has 679, ClaudeBot 592. As of 13 October, none of our articles published or updated more than seven days earlier had been missed by a verified AI indexing crawler. One mistake remained on our side: our article already described the forgotten Anthropic range, but our script still ignored it and at first classed 670 ClaudeBot visits as spoofed. It now reads the list published by Anthropic at every check. The Apple, Amazon and Diffbot crawlers, for their part, are still cut off by the host’s firewall.

## Why would an AI crawler allowed in robots.txt stay outside?

robots.txt is a sign on the door, read by the crawlers that choose to read it. It gives no orders to your infrastructure. Before reaching your page, a request often passes through several layers: a CDN if you have one, a web application firewall, and with some hosts a bot protection shared by all their customers. Each applies its own rules, set somewhere other than your robots.txt, and each can turn away a crawler that robots.txt welcomes.

A 403 refusal at least leaves a line in your logs. On our site, the host’s firewall dropped the connection without answering at all, with no status code and no line in the site’s log. Any request whose identifier contained “Applebot”, “Amazonbot” or “Diffbot” was shown the door, while the same request without that word went through. Our robots.txt had been inviting them since the summer: not one visit in six weeks, and no warning. Replayed on 30 September and again on 13 October, those requests are still cut off, and lifting the block is down to the host. It is the same family of failure as [tracking that lies in silence](https://dotsland.com/en/tracking-server-side-lying-in-silence/): nothing breaks on screen, the data is simply missing.

Here is, for each layer, what it decides and what it keeps out of your sight.

| Layer | What it does | What it does not tell you |
| --- | --- | --- |
| **robots.txt** | Tells the crawlers that respect it what they may read. User-triggered agents may ignore it | Whether the crawler actually got in, and whether it was really that crawler |
| **Your CDN rule** | Allows, blocks or challenges crawlers, often by categories set once and forgotten | What it turned away, until you open its event log |
| **Web application firewall, host protection** | Filters requests by signature, sometimes on the crawler’s name alone | Nothing at all when it drops the connection without answering, which is what happened to us |
| **Access logs** | Show who came, from which address, with which response code and on which day | Who was stopped before reaching the server |
| **Replaying the identifier** | Reveals in one command that a layer refuses a name | Whether the real crawler, from its real addresses, is treated the same way |

## The AI crawler families to look for in your logs

Every AI company sends out several crawlers. Their documentation lets you sort them, as we read it, by family, which tells you which ones to look for and how to read their visits.

GPTBot at OpenAI and ClaudeBot at Anthropic are **training crawlers**: they collect content that may be used to train models. So does Amazonbot, and Applebot also powers search in Siri, Spotlight and Safari. OpenAI states that its settings are independent: blocking GPTBot does not take you out of its search.

OAI-SearchBot, Claude-SearchBot and PerplexityBot are **search crawlers**: they surface sites in the search results of ChatGPT, Claude and Perplexity, and Perplexity adds that its crawler is not used to train models. As we read it, these are the ones that matter most if you want to be cited in an answer. Being read is only the first condition: citation then depends, to a large extent, on what others write about you, as our [AI citability audit](https://dotsland.com/en/ai-citability-audit/) shows.

ChatGPT-User, Claude-User and Perplexity-User are **user-triggered agents**: they fetch a page because a person asked for it. According to OpenAI, robots.txt rules may not apply to ChatGPT-User; Perplexity and Google say their agents of this kind generally ignore them. Anthropic says its bots respect robots.txt. These visits measure one person’s use, not a search engine’s interest.

Two names you often see in robots.txt files are not crawlers at all. Google-Extended and Applebot-Extended are tokens that say whether your content may be used for Google’s or Apple’s AI. Google states that Google-Extended has no user agent of its own, and Apple that Applebot-Extended does not crawl any page. You will never see them in your logs, and that is normal.

## Which robots.txt rules affect your AI visibility, and how do you write them?

The answer depends on what you want, and the settings are independent: OpenAI states that blocking GPTBot does not remove you from its search. Three decisions, to be taken separately.

| Your goal | The names concerned | What to write |
| --- | --- | --- |
| **Being cited in AI engine answers** | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Allow them |
| **Refusing that your content trains models** | GPTBot, ClaudeBot, and the Google-Extended and Applebot-Extended tokens | Disallow them, with no effect on search crawlers |
| **Blocking fetches requested by a user** | ChatGPT-User, Claude-User, Perplexity-User | Robots.txt is not always enough: according to their publishers, several of these agents may ignore it |

At Dotsland we allow everything: our articles are made to be read and cited. A brand protecting high-value content can do the opposite on training without giving up search. Here is an example block that welcomes search and refuses training:

```
# AI engine search: welcome
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /wp-admin/

# Model training: refused
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
```

One trap comes up often. The robots.txt standard, RFC 9309, says a crawler follows the group that carries its name, and only falls back on the `User-agent: *` group when there is no such group. As soon as you name a crawler, the general rules written for everyone therefore no longer apply to it: that is why the example repeats `Disallow: /wp-admin/` in the search crawlers’ group. Re-read your named groups after every change.

## How do you check your crawlers, one by one?

> **To see where you stand**  
> Of the crawlers your robots.txt names, which ones appear at least once in this month’s logs? For the most frequent of them, what share of its visits comes from its official addresses? And your latest article: which verified crawler read it, and how many days after publication? If you cannot answer any of these questions, start by asking your host for your log export.

To answer them, we use the following method, with a text editor or a spreadsheet.

1. Get thirty days of access logs from your hosting control panel or your host’s support team. Archive them straight away: with our host, the August archive, read on 12 September, was gone by the 24th.
2. Filter by crawler name. At a minimum, look for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Applebot and Amazonbot. Set the user-triggered agents aside.
3. Check the address of every visit. OpenAI, Anthropic, Perplexity, Google, Apple and Amazon publish theirs, with links in our sources. For Google, a reverse DNS lookup followed by a lookup of the resulting name must give back the same address, or the address must appear in its range files. A visit from outside the ranges is fake until proven otherwise. If a whole crawler falls outside, reread the list first, and note which list you used and how old it is.
4. Count the response codes, one verified crawler at a time. 200, the page was served. 403, the server understood and refuses. 429, the crawler sent too many requests and is being throttled. 503, the server says it is unavailable for now. A genuine crawler piling up 403s or 429s is being stopped by a layer robots.txt does not control.
5. Look for the absentees. For an invited crawler that never shows up, replay its identifier, for instance with `curl -I -A "Applebot" https://your-site.co.uk/`, then the same command with another name. A connection dropped without an answer points to a protection upstream of the site. This test only covers the name: the logs tell you what the real crawler, from its real addresses, actually received.
6. Look at the dates, not the totals. For each recent article, note the first visit by a verified crawler and the delay since publication, and keep that history from one month to the next. That is how we saw GPTBot stop despite a flattering August total.

## A block is lifted in the layer that set it

A site’s access policy is written in several places that do not talk to each other, and a CDN rule set by a supplier or a protection a host applies to all its customers can contradict what the business decided in its robots.txt. The fix therefore has to be requested from the layer that refuses. For a genuine crawler piling up 403s or 429s, open the event log of the CDN or the web application firewall and look for the rule that targets it. For a connection dropped without an answer, write to your host’s support with the test command and its result.

Once the rule has changed, replay the command, then check in the following month’s logs that the real crawler, from its addresses, now gets 200s and reads your new articles.

If you would rather not run this check alone, we can read your server logs and your CDN settings with you, then find the layer that stops the crawlers you invite. [Ask for a log reading session](https://dotsland.com/en/contact-us/): it is part of our [analytics and performance expertise](https://dotsland.com/en/expertise/analytics-and-performance/).

## Sources

Each source was opened on 30 September 2026, the day we also replayed our access tests.

1. OpenAI, “Overview of OpenAI Crawlers”, and the IP range files for GPTBot, OAI-SearchBot and ChatGPT-User. [View](https://developers.openai.com/api/docs/bots)
2. Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”, Claude Help Center. [View](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
3. Anthropic, list of its crawlers’ addresses, file dated 18 August 2026. [View](https://claude.com/crawling/bots.json)
4. Perplexity, “Perplexity Crawlers”, with the address files for PerplexityBot and Perplexity-User. [View](https://docs.perplexity.ai/guides/bots)
5. Google, “Verify Requests from Google Crawlers and Fetchers”, updated 20 March 2026. [View](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
6. Google, “Google’s common crawlers”, Google-Extended section, updated 14 July 2026. [View](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
7. Google, “Google User-Triggered Fetchers”, updated 19 August 2026. [View](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers)
8. Apple, “About Applebot”, 4 September 2026. [View](https://support.apple.com/en-us/119829)
9. Amazon, “About AmazonBot”. [View](https://developer.amazon.com/amazonbot)
10. IETF, RFC 9110, “HTTP Semantics”, codes 403 and 503, and RFC 6585, “Additional HTTP Status Codes”, code 429. [View](https://www.rfc-editor.org/rfc/rfc9110)
11. IETF, RFC 9309, “Robots Exclusion Protocol”, section 2.2.1 (group applicable to a crawler), read on 13 October 2026. [View](https://www.rfc-editor.org/rfc/rfc9309)

## FAQ

### Should I block training crawlers and keep the search crawlers?

You can with OpenAI, which states that each setting is independent, and with Google, where Google-Extended has no effect on your inclusion in Search or on your rankings. The decision depends on your business model. In our view, for an SME selling products or services, being read by search crawlers matters more than the rest. Either way, check afterwards in your logs that the rule has the effect you intended.

### Why did ChatGPT read my page when robots.txt blocks its crawlers?

Most likely because a user asked it to. ChatGPT-User, Perplexity-User and Google’s user-triggered fetchers are documented as able to ignore robots.txt. Check the address of the visit: if it is on the company’s list, it really was their agent, sent at someone’s request.

### Is blocking a crawler’s IP addresses safer than robots.txt?

Not necessarily. Anthropic warns that blocking by address may not guarantee a lasting opt-out, because it also stops its crawler from reading your robots.txt. And the ranges change: the Anthropic list we read was dated 18 August 2026. robots.txt remains the tool designed to state your choice, and the logs are there to check it is respected.