Every week an AI crawler from OpenAI, Anthropic, Perplexity, Google, or ByteDance almost certainly hits your website, and Google Analytics shows you none of it. I run AI-visibility audits for medspa and clinic sites, and the first question owners ask is always the same: “which AI bots are actually crawling my site?” This is the directory and log-reading guide I use to answer it: every major AI user agent, where the evidence lives, and how to tell real bots from spoofers.
Why Google Analytics can’t see AI crawlers
Google Analytics (and almost every other analytics tool) works by running a JavaScript snippet in the visitor’s browser. AI crawlers don’t run JavaScript for tracking purposes. Most of them request your raw HTML, take what they need, and leave. No JS execution means no GA4 event, which means your analytics dashboard reports zero AI bot visits even if GPTBot fetched 400 pages last night.
The evidence lives in exactly one place: your server access logs (or your CDN’s request logs, if you’re behind Cloudflare or similar). Every request that reaches your server gets a log line with the visitor’s IP address and its self-declared user agent string. That’s where AI crawlers identify themselves, and that’s the file we’re going to read.
The scale of what you’re missing is not trivial. Cloudflare’s “From Googlebot to GPTBot” report found GPTBot’s share of verified bot traffic across its network jumped from 4.7% in July 2024 to 11.7% in July 2025, with ClaudeBot growing from roughly 6% to nearly 10% over the same period. AI crawlers are now a meaningful slice of all bot traffic on the internet, and they’re crawling small local-business sites, not just news publishers.
The directory: AI crawler user agents in 2026
Here is every AI user agent I look for when I audit a client’s logs. The “purpose” column matters more than anything else on this page. We’ll get to why in a moment.
| User agent token | Operator | Purpose | Type |
|---|---|---|---|
| GPTBot | OpenAI | Collects content for training OpenAI’s foundation models | Training crawler |
| OAI-SearchBot | OpenAI | Indexes content so it can appear (with links) in ChatGPT search results | Search/index crawler |
| ChatGPT-User | OpenAI | Fetches a page live when a ChatGPT user asks about it or clicks browse | User-triggered fetcher |
| ClaudeBot | Anthropic | Collects training data for Claude models | Training crawler |
| Claude-SearchBot | Anthropic | Indexes content to improve Claude’s search result quality | Search/index crawler |
| Claude-User | Anthropic | Fetches pages live when a Claude user asks a question | User-triggered fetcher |
| PerplexityBot | Perplexity | Indexes content for Perplexity’s answer engine (surfaced with citations) | Search/index crawler |
| Perplexity-User | Perplexity | Fetches a page live during a user’s Perplexity query | User-triggered fetcher |
| Google-Extended | robots.txt control token for Gemini training (no separate crawler; regular Googlebot does the fetching) | Training (control token) | |
| Bytespider | ByteDance (TikTok) | Training data collection; widely reported to ignore robots.txt | Training crawler |
| CCBot | Common Crawl | Builds the open Common Crawl dataset used to train many AI models | Training crawler |
| Amazonbot | Amazon | Crawling for Alexa and Amazon AI answers | Mixed crawler |
| Applebot-Extended | Apple | Control token for Apple Intelligence training; standard Applebot does the crawling | Training (control token) |
| meta-externalagent | Meta | Training data collection for Meta’s Llama models | Training crawler |
Full user agent strings are longer than the tokens above. For example, per OpenAI’s official crawler documentation, GPTBot identifies as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot. In practice you only need to match the token. Any log line containing “GPTBot” is (claiming to be) OpenAI’s training crawler.
The distinction that actually matters: training vs. search vs. user-triggered
Most articles treat “AI bots” as one category. That’s how site owners end up blocking everything and then wondering why their clinic never gets cited in ChatGPT. There are three distinct behaviors, and blocking each has completely different consequences:
- Training crawlers (GPTBot, ClaudeBot, Bytespider, CCBot, meta-externalagent). These collect content to train future models. Blocking them means your content stays out of the next model version’s training data. It does not remove anything already collected, and it does not stop you from appearing in AI search answers.
- Search/index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). These build the retrieval indexes that AI answer engines cite from. Block these and you are opting out of being a cited source in ChatGPT search, Claude, and Perplexity answers. For a local business trying to get recommended when someone asks “best medspa near me,” that’s usually the opposite of what you want.
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User). These fire when a real human, in a real chat session, asks the AI about your site right now. This is the closest thing to a live prospect in your AI-crawl data. Blocking these means the AI tells that user “I couldn’t access the site” while your competitor’s page loads fine.
OpenAI’s documentation is explicit that these controls are independent: you can disallow GPTBot (no training) while allowing OAI-SearchBot (yes citations). Anthropic’s crawler documentation confirms the same three-way split, and states all three of its bots honor robots.txt. This is the whole reason identifying which bots hit your site matters: it tells you which decision you’re actually making. For the blocking side of that decision, I’ve written a separate robots.txt guide for AI bots with copy-paste configurations.
How to check manually: grep your access logs
If you’re comfortable with a terminal, one command answers the question. On any Apache or nginx server, run this against your access log:
grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|perplexitybot|perplexity-user|bytespider|ccbot|amazonbot|meta-external" access.log | awk '{print $1, $12, $13, $14}' | sort | uniq -c | sort -rn | head -40For a quick per-bot count:
grep -ioE "gptbot|claudebot|perplexitybot|bytespider|ccbot|amazonbot" access.log | sort | uniq -c | sort -rnWhere to find the logs on common hosts
- Hostinger: hPanel → Websites → Dashboard → Analytics/Logs, or via SSH the logs sit under
~/logs/(older entries are gzipped, so usezgrepon those). - SiteGround: Site Tools → Statistics → Access Log, or download raw logs via SSH/SFTP from the
logsdirectory in your account root. - WP Engine: User Portal → your site → Logs → Access logs (viewable and downloadable per environment; they retain a rolling window, so check regularly).
- Cloudflare: if your DNS is proxied through Cloudflare, the dashboard’s Security → Bots and the AI Audit / AI Crawl Control section show AI crawler hits by name without touching server logs at all. This is the easiest option and includes bot verification built in.
The 60-second way: paste your log into my free analyzer
If grep isn’t your thing, I built a free AI bot log analyzer that does all of this in your browser. Download a slice of your access log, paste it in, and it identifies every AI crawler, groups hits by operator and purpose (training vs. search vs. user-triggered), and flags user agents that don’t match known patterns.
Privacy note: the analyzer runs entirely client-side. Your log file never leaves your browser and never touches my server, which matters because access logs contain your visitors’ IP addresses.
Verifying real bots vs. spoofers
A user agent string is just text. Any scraper can send “GPTBot” in its headers, and plenty of aggressive scrapers do exactly that to borrow the reputation of legitimate crawlers. Before you make decisions based on log data (and especially before you allowlist anything in a firewall), verify against the operators’ published IP ranges:
- OpenAI publishes machine-readable IP lists at
openai.com/gptbot.json,openai.com/searchbot.json, andopenai.com/chatgpt-user.json. - Perplexity publishes ranges at
perplexity.com/perplexitybot.jsonandperplexity.com/perplexity-user.json, per Perplexity’s crawler docs. - Google and Apple support reverse-DNS verification:
host <IP>should resolve to a googlebot.com/google.com or applebot.apple.com hostname, and the forward lookup of that hostname should return the same IP. - Bytespider publishes no verification ranges, which is one of several reasons most operators treat it differently from the Western crawlers.
A request claiming to be GPTBot from an IP outside OpenAI’s published ranges is a spoofer, and you can block it without any citation consequences.
Should you block or allow? A framework by business type
Here’s how I think about it, and Cloudflare’s own data explains why nuance matters. Their crawl-to-click analysis found Anthropic’s crawlers made roughly 38,000 requests per referred human visitor as of July 2025 (down from about 286,000:1 in January 2025), while Perplexity sat near 194:1; for news sites the ratios were far lower (Anthropic ~2,500:1, OpenAI ~152:1, Perplexity ~33:1). Translation: AI crawlers take a lot and send back few clicks, but the clicks and citations they do send are increasingly how buyers discover businesses.
- Local service businesses (medspas, clinics, agencies): allow search/index crawlers and user-triggered fetchers. Your entire goal is to be the answer when someone asks an AI “who does lip filler in [city]”. That’s the core of answer engine optimization. Training crawlers are a judgment call; I usually allow them because the visibility upside outweighs any content-protection concern for marketing pages.
- Publishers and content businesses: your content is the product, so blocking training crawlers while allowing search bots is a defensible middle path.
- Sites with proprietary data (pricing engines, member content, original research): block training crawlers, consider blocking everything from operators that don’t publish verification ranges, and rate-limit the rest.
What healthy AI-crawl activity looks like
Benchmarks here are rough because operators change crawl behavior monthly. Cloudflare’s data shows GPTBot and ClaudeBot repeatedly swapping rank through 2025–2026. But from the medspa and local-business logs I review, a directionally healthy pattern looks like this (est., for a typical 30–150 page local site):
- GPTBot / ClaudeBot: anywhere from a handful to a few hundred requests per month, usually in bursts that re-crawl the whole site (est.).
- OAI-SearchBot / PerplexityBot / Claude-SearchBot: steadier, lower-volume crawling of your key pages; seeing these regularly is a good sign your site is in the retrieval indexes (est.).
- ChatGPT-User / Claude-User / Perplexity-User: sporadic single-page hits. Each one is a live human asking an AI about you. If these are trending up, your generative engine optimization work is doing its job.
- Red flags: thousands of Bytespider hits per day, “GPTBot” requests from unverifiable IPs, or crawl bursts hammering your booking pages hard enough to slow the site for real patients. That’s when you rate-limit or block.
The zero-signal case matters too: if your logs show no AI search bots at all, check whether a security plugin or CDN rule is blocking them silently. I’ve audited medspa marketing setups where an overzealous firewall rule had been quietly excluding the site from every AI answer engine for months. And once you’ve confirmed the right bots are getting in, make sure they can understand what they find, my free llms.txt validator checks the file that tells AI systems what your site is about.
Want a second set of eyes on this for your clinic? Book a free strategy call or call/text me at +91 97297 12388.
Frequently asked questions
Why don’t AI bots show up in Google Analytics?
GA4 requires JavaScript to execute in the visitor’s browser to record a hit. AI crawlers request your HTML directly and don’t run tracking scripts, so they’re invisible to analytics tools. The only reliable record is your server access log or your CDN’s request logs.
What’s the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI’s training crawler: it collects content to train future models on its own schedule. ChatGPT-User fires only when a real person in a ChatGPT session asks about your page right then. Blocking GPTBot keeps you out of training data; blocking ChatGPT-User breaks the experience for a live prospect trying to look at your site through ChatGPT.
How do I know a bot claiming to be GPTBot is really from OpenAI?
Check the request’s IP address against OpenAI’s published ranges at openai.com/gptbot.json. Perplexity publishes similar JSON files, and Google and Apple support reverse-DNS verification. Any “GPTBot” request from an IP outside the published ranges is a spoofer wearing a costume.
Will blocking AI training bots hurt my rankings or AI citations?
Blocking training crawlers like GPTBot or ClaudeBot has no effect on Google rankings and does not stop you from being cited in AI search answers. Citations come from the search/index bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking those search bots, however, does opt you out of AI citations, which for most local businesses is a mistake.
Do AI crawlers respect robots.txt?
OpenAI, Anthropic, Google, Apple, Amazon, and Perplexity’s index crawlers all document robots.txt compliance; Anthropic even supports the Crawl-delay directive. Bytespider (ByteDance) is widely reported to ignore robots.txt, which is why controlling it usually requires firewall or CDN rules rather than a polite request.
How often should I check my logs for AI crawler activity?
Monthly is enough for most local businesses. Crawler behavior shifts fast (Cloudflare’s data shows the leading AI crawlers trading places repeatedly through 2025 and 2026), so a monthly pass through my log analyzer catches new bots, spoofing attempts, and any accidental blocks before they cost you months of AI visibility.
