robots.txt for AI Bots — The 2026 Playbook
robots.txt in 2020 was a Googlebot file. robots.txt in 2026 is an AI-citation control panel with 25-plus user-agents to think about, a CDN layer that silently overrides what you wrote, and a one-character typo that can wipe your brand off ChatGPT or Claude. I have rewritten robots.txt files for 40-plus client sites in the last six months. This is the 2026 playbook with five copy-paste templates by business type, the CDN trap that breaks them, and the exact decisions I make on every audit.
What changed since 2020
Three things.
First, the number of meaningful user-agents went from roughly five to roughly 25. Googlebot, Bingbot, Yandexbot, Baiduspider, and DuckDuckBot covered the old surface. Today you also need to think about OpenAI’s GPTBot, OAI-SearchBot, and ChatGPT-User. Anthropic’s ClaudeBot, Claude-SearchBot, Claude-User, and anthropic-ai. Perplexity’s PerplexityBot and Perplexity-User. Apple’s Applebot and Applebot-Extended. Google’s Google-Extended and Google-NotebookLM and Google-CloudVertexBot and Google-Agent. Plus Meta’s meta-externalagent and FacebookBot, ByteDance’s Bytespider, Mistral’s MistralAI-User, Amazon’s Amazonbot, DuckDuckGo’s DuckAssistBot, and Common Crawl’s CCBot. Each behaves differently. Each demands a decision.
Second, the consequence of getting it wrong moved from a small SEO impact to a brand-invisibility impact. In 2020, blocking the wrong bot might dent organic rankings on a long-tail query. In 2026, blocking OAI-SearchBot removes you from ChatGPT search entirely. Blocking ClaudeBot removes training-data familiarity. Blocking Claude-SearchBot removes you from claude.ai citations. The same is true for Perplexity. The blast radius is much larger.
Third, the CDN layer became the primary point of failure. Cloudflare’s AI Bot blocking toggle, Vercel’s WAF rules, Fastly’s bot fight modes, and AWS WAF rules all sit in front of robots.txt. Your file can say allow and the CDN can return 403 before the bot ever sees your file. I audit the CDN layer before I read the robots.txt file on every accessibility audit because the CDN is what is actually answering the request.
The five categories of AI user-agents
This is the framework I use to make decisions deterministically.
- Training crawlers. Fetch content to train the underlying LLM. Examples: GPTBot, ClaudeBot, CCBot, meta-externalagent. These do not drive citations directly. They influence what the LLM “knows” about your brand from its training data, which has a long-term influence on recall.
- Search and retrieval crawlers. Build the indexes that AI engines use to answer live queries. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot, anthropic-ai. These drive citations. Blocking them is what removes you from the corresponding AI engine.
- User-triggered fetchers. Fetch a specific URL only when a human asks the AI to. Examples: ChatGPT-User, Claude-User, Perplexity-User, Google-NotebookLM, Applebot, MistralAI-User. These drive citations on live queries and need to be allowed unless you have a specific reason not to.
- Opt-out tokens. Signal training preferences without doing the crawling themselves. Examples: Google-Extended for Gemini and Bard, Applebot-Extended for Apple Intelligence. Blocking these signals “do not use my content to train” without blocking the underlying search crawler.
- Embedded agents. IDE and desktop assistants that read content on demand. Examples: Cursor’s crawler, Cline, Continue, Claude Desktop’s MCP fetchers. Increasingly important for B2B SaaS and developer-tooling sites.
Pick the right action per category. Training crawlers: allow for service businesses, conditional for publishers and content sellers. Search and retrieval crawlers: always allow. User-triggered fetchers: always allow. Opt-out tokens: allow for visibility, block if you object to model training. Embedded agents: always allow.
The full allow and block matrix
⚡ 2-minute scorecard · instant result
How strong is your lead engine?
Answer 5 quick questions. Get your score + the top fixes — free.
1. Do you track which source every lead comes from?
2. Do you respond to new leads in under 5 minutes?
3. Do you have a CRM that catches every inquiry?
4. Do you run a follow-up / nurture sequence?
5. Is your site built to convert, not just inform?
| Bot | Operator | Category | Citation value | Default action |
|---|---|---|---|---|
| Googlebot | Search + AI Overviews | High | Never block | |
| Bingbot | Microsoft | Search + Copilot | High | Always allow |
| Applebot | Apple | Siri + Apple Intelligence | Medium | Always allow |
| OAI-SearchBot | OpenAI | ChatGPT search | High | Always allow |
| ChatGPT-User | OpenAI | Live ChatGPT queries | High | Always allow |
| GPTBot | OpenAI | Training | None direct | Allow for service business |
| Claude-SearchBot | Anthropic | Claude citations | High | Always allow |
| Claude-User | Anthropic | Live Claude queries | High | Always allow |
| anthropic-ai | Anthropic | Public search | Medium | Allow |
| ClaudeBot | Anthropic | Training | None direct | Allow unless content-sensitive |
| PerplexityBot | Perplexity | Search, always cites with click-through | Highest per crawl | Always allow |
| Perplexity-User | Perplexity | Live queries | High | Always allow |
| Google-Extended | Opt-out token, Gemini training | Medium | Allow if Gemini visibility matters | |
| Google-NotebookLM | User-triggered | Medium | Allow | |
| Google-Agent | Gemini agent | Medium | Allow | |
| DuckAssistBot | DuckDuckGo | Search assistant | Medium | Allow |
| Amazonbot | Amazon | Alexa, Rufus | Medium | Allow |
| MistralAI-User | Mistral | User-triggered | Limited | Allow |
| Applebot-Extended | Apple | Opt-out for Apple training | None direct | Block if you object to Apple training |
| meta-externalagent | Meta | Training, high volume | None direct | Block unless you want Meta AI features |
| FacebookBot | Meta | Training | None | Block |
| Bytespider | ByteDance, TikTok | Training, aggressive bandwidth | None | Block |
| CCBot | Common Crawl | Training, feeds many vendors | None direct | Block |
| Diffbot | Diffbot | Commercial scraping | None unless customer | Block |
The defaults above are the right starting point for 90% of service businesses. The remaining 10% need vertical-specific tuning, covered in the templates below.
My full AI bot reference has the operator details, IP ranges, and identification tips for each bot if you need to verify them in server logs.
Template 1: Service business
This is my default template for medspa, agency, consulting, legal, accounting, and any service business that wants maximum AI visibility.
# Sprout Sage Solutions robots.txt — service business 2026
# Standard search engines — never block
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: Applebot
Allow: /
# AI search and user-triggered — these drive citations
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Google-NotebookLM
Allow: /
User-agent: DuckAssistBot
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: MistralAI-User
Allow: /
# Training crawlers — allow, you want LLMs to know your brand
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
# Block high-volume, zero-value scrapers
User-agent: Bytespider
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: CCBot
Disallow: /
# Default rules for all other bots
User-agent: *
Disallow: /wp-admin/
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /?s=
Disallow: /search/
Sitemap: https://sproutsagesolutions.com/sitemap.xml
Why this works for service businesses: you want your brand to show up in ChatGPT search, Claude citations, and Perplexity answers. You also want LLMs trained on your content so future model versions recall your brand. The training crawlers cost you nothing in lost traffic and earn long-term recall. The zero-value scrapers like Bytespider hammer bandwidth without ever citing you, so blocking them protects your hosting bill.
Template 2: Ecommerce
Ecommerce robots.txt needs everything in the service business template plus aggressive faceted-nav control. The biggest SEO mistake on ecommerce sites is letting AI crawlers index thousands of thin faceted URLs.
# Ecommerce robots.txt — Shopify or WooCommerce 2026
# AI search and user-triggered
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot
Allow: /
# Training crawlers — allow for brand recall in product recommendations
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
# Block scrapers
User-agent: Bytespider
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: CCBot
Disallow: /
# Default — block thin and duplicate URLs
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /my-account/
Disallow: /?orderby=
Disallow: /?filter_*
Disallow: /?add-to-cart=
Disallow: /?wc-ajax=
Disallow: /?s=
Disallow: /search?
Disallow: /tag/
Disallow: /author/
Disallow: /wp-admin/
Sitemap: https://yourstore.com/sitemap.xml
Sitemap: https://yourstore.com/product-sitemap.xml
For Shopify the platform builds most of the disallow rules automatically, but Shopify added editable robots.txt back in 2021 so you can customize. I add explicit Allow rules for the AI bots because Shopify’s default does not include them.
For WooCommerce the explicit Disallow rules for /?orderby= and /?filter_* are essential. WooCommerce filtering creates thin URLs that crawl budgets get wasted on. Blocking these from all bots, AI and traditional, keeps the crawl focused on real product pages.
Template 3: Publisher
Publishers are the one category where blocking training crawlers is defensible. You sell content. Allowing GPTBot or ClaudeBot to train models on your work without compensation is a real revenue question.
# Publisher robots.txt 2026
# Standard search — never block
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: Applebot
Allow: /
# AI search — allow, these drive citations and referral traffic
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-NotebookLM
Allow: /
# Block training crawlers — your content is your product
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: FacebookBot
Disallow: /
# Default
User-agent: *
Disallow: /wp-admin/
Disallow: /?s=
Sitemap: https://yourpublication.com/sitemap.xml
This is the publisher-specific play. Block training, allow search. The reasoning is that OAI-SearchBot drives citation and referral traffic, so blocking it would cost real money. GPTBot is training only, so blocking it costs nothing in the current traffic mix and prevents OpenAI from training Gemini or GPT-5 on your archive without compensation.
The New York Times, Reuters, and several major publishers have implemented variants of this. The legal cases against OpenAI and Anthropic over training data have pushed publishers toward this posture even before any judgement.
Template 4: SaaS or developer tooling
SaaS and developer tooling have a different priority: the docs need to be readable by IDE agents and MCP doc servers. The marketing site needs to be readable by AI search. The dashboards need to be off-limits.
# SaaS / developer tooling robots.txt 2026
# Standard search
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: Applebot
Allow: /
# AI search and user-triggered — all allowed for docs and marketing
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Google-NotebookLM
Allow: /
User-agent: MistralAI-User
Allow: /
# Training crawlers — allow, you want LLMs to know your API
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
# Block scrapers
User-agent: Bytespider
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: CCBot
Disallow: /
# Default — block app surfaces
User-agent: *
Disallow: /app/
Disallow: /dashboard/
Disallow: /api/internal/
Disallow: /admin/
Disallow: /login
Disallow: /signup
Disallow: /?s=
Sitemap: https://yourapp.com/sitemap.xml
Sitemap: https://docs.yourapp.com/sitemap.xml
The SaaS-specific edge: developers ask Claude and ChatGPT how to integrate with your API. If your docs site is blocked from training crawlers, the LLM does not know how your API works and recommends a competitor whose docs are accessible. For SaaS, allowing training is strategic, not optional.
Template 5: Content seller, course, or membership site
If you sell the content itself, the calculus shifts. Allowing training crawlers to ingest your gated content is a revenue leak. Allowing search crawlers to index your marketing pages is still valuable.
# Content seller robots.txt 2026
# Standard search
User-agent: Googlebot
Allow: /
Disallow: /members/
Disallow: /lessons/
Disallow: /private/
User-agent: Bingbot
Allow: /
Disallow: /members/
Disallow: /lessons/
# AI search — allow marketing pages, block gated content
User-agent: OAI-SearchBot
Allow: /
Disallow: /members/
Disallow: /lessons/
Disallow: /private/
User-agent: ChatGPT-User
Allow: /
Disallow: /members/
User-agent: Claude-SearchBot
Allow: /
Disallow: /members/
User-agent: PerplexityBot
Allow: /
Disallow: /members/
# Block ALL training crawlers — your content is your product
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
# Default
User-agent: *
Disallow: /members/
Disallow: /lessons/
Disallow: /wp-admin/
Sitemap: https://yoursite.com/sitemap.xml
This is the most restrictive of the five templates. The marketing pages stay open for AI search so prospects can find you. The gated content is blocked from every bot including standard search engines because there is no SEO benefit to indexing content that requires login.
If your members area is behind authentication anyway, the robots.txt rules are belt-and-suspenders. Bad actors will ignore them. Compliant crawlers will respect them.
The CDN trap, in detail
This is the single most common cause of AI bots not crawling sites that have correct robots.txt files. The robots.txt file says allow. The CDN says no. The CDN wins because it answers the request first.
Cloudflare’s AI Bot blocking is the most visible offender. Cloudflare ships a one-toggle “Block AI Bots” feature that blocks 20-plus AI user-agents at the edge with a 403 response. The toggle was enabled by default for free-tier sites for a period in 2024, which left thousands of sites with locked-down AI access without their owners knowing.
How to verify your CDN is not overriding your robots.txt:
- Fetch your robots.txt with a User-Agent header set to OAI-SearchBot.
curl -A "OAI-SearchBot/1.0" https://yourdomain.com/robots.txt. If the response is 403, the CDN is blocking before robots.txt is read. - Fetch a real page with the same User-Agent.
curl -A "OAI-SearchBot/1.0" https://yourdomain.com/. If this is 403 while the browser fetch works, the CDN is the problem. - Check the Cloudflare dashboard under Security, Bots, AI Crawlers and Scrapers. If the toggle is enabled, disable it and add explicit rules for the bots you want to allow.
- If you use Vercel, check the Web Application Firewall and Bot Management settings. Vercel’s default rules sometimes flag AI crawlers as suspicious traffic.
- If you use Fastly or AWS WAF, ask your platform team for the bot rule set and check it against the allow matrix above.
I run this verification on every AI Accessibility Audit. About one in three audits surfaces a CDN-level block that the client did not know about.
Rate-limiting that backfires
Aggressive rate-limiting on AI crawler IP ranges has the same effect as blocking them in robots.txt. The crawler retries, gets throttled, retries fewer pages next time, and eventually crawls less frequently.
The pattern I see: a hosting provider or CDN flags AI bots as suspicious because their crawl pattern looks like a scrape. The provider throttles to 1 request per 10 seconds. The bot backs off. Citation rate drops three months later because the crawler is now visiting once a week instead of twice a day.
The fix: explicit rules that exempt AI bots from rate limiting up to a sane budget. 5 to 10 requests per second is generous and well below the volumes most AI crawlers actually use. If your bandwidth genuinely cannot handle that, the underlying issue is hosting capacity, not bot behavior.
How to know it is working
Three signals to monitor after deploying or rewriting robots.txt.
Server access logs. Within 7 to 14 days you should see AI bot User-Agents in the logs at higher frequency than before. ClaudeBot, OAI-SearchBot, and PerplexityBot are the high-signal ones. If you see zero hits from these three over 30 days post-deploy, the CDN is probably still blocking them.
Citation monitoring. Otterly, AthenaHQ, or Profound tracking your target prompts. The first new citations show up at 30 to 60 days. Citation count growth and share-of-voice gains are the leading indicators.
LLM referral traffic. GA4 custom channel grouping for chatgpt.com, perplexity.ai, claude.ai, gemini.google.com, and copilot.microsoft.com. Volume is small at first but the conversion rate on this traffic is typically 2 to 4 times higher than blended organic. Track sessions, conversions, and revenue per session separately.
I bundle all three signals into the monthly report for GEO retainer clients. The report shows AI bot hit frequency, citation count by engine, share-of-voice deltas, and LLM referral traffic in the same view so the trend is visible at a glance.
The combined stack: robots.txt plus llms.txt plus schema
robots.txt alone is the floor. The stack that actually moves AI citation is the four-part bundle.
robots.txt permits AI crawlers and blocks zero-value scrapers. This is the access-control layer.
llms.txt at the root tells AI crawlers which pages are most important. This is the curation layer. My complete 2026 llms.txt guide covers the spec, the worked example, and the auto-generators.
Schema markup on every cornerstone page. Article plus FAQPage plus BreadcrumbList plus Person plus Organization. BrightEdge data shows pages with three to four schema types are cited twice as often as single-schema pages. This is the parsing layer.
Content shaping per the Princeton GEO playbook. Inline citations, statistics density, FAQ depth, answer-first openings. This is where the real citation lift lives.
I deliver all four together as my AI Accessibility Audit, a $300 one-time package that delivers in five business days. It covers the robots.txt rewrite, the llms.txt build, the schema audit and implementation pack for the top 5 pages, and the AI citation baseline across 10 target queries.
The mistakes I see on every audit
Five robots.txt mistakes I find on roughly half the sites I audit.
1. Blanket-blocking everything with “Bot” in the name
The owner saw a list of AI bots, panicked, and pasted Disallow: / for anything with bot in the name. This kills OAI-SearchBot alongside GPTBot. The brand vanishes from ChatGPT search entirely. The fix is the matrix above: block specific scrapers by name, allow search and user-triggered bots explicitly.
2. Blocking GPTBot then wondering why ChatGPT does not recommend the brand
The most common 2026 mistake. GPTBot is training. OAI-SearchBot is search. Blocking GPTBot has zero impact on whether ChatGPT recommends you today. Blocking OAI-SearchBot is what makes you invisible. The names sound similar enough that owners block the wrong one. My GPTBot decision framework walks through the full reasoning if you want the long version.
3. CDN overriding robots.txt
Covered in detail above. Cloudflare’s AI Bot toggle is the usual culprit. Always verify with curl plus the bot’s User-Agent before trusting the file.
4. Aggressive rate-limiting on AI crawler IPs
Also covered above. The bot backs off, citations drop, the owner blames the content.
5. Forgetting that robots.txt is a request, not a firewall
Robots.txt only stops compliant crawlers. Bad actors ignore it. If your goal is actual content protection, you need IP rules, authentication, or paywalls. I see sites block GPTBot in robots.txt and call it secure. The same content is being scraped by 50 non-compliant crawlers a day that ignore the file entirely.
What to do this week
Three actions, in order.
1. Verify your current robots.txt does what you think it does. Fetch it with curl plus the User-Agent for OAI-SearchBot, ClaudeBot, and PerplexityBot. If any returns 403, the CDN is the problem and the file does not matter until the CDN is fixed.
2. Rewrite the file using the template above that matches your business type. Service business, ecommerce, publisher, SaaS, or content seller. 30 minutes of work. Test the new version against the same curl commands. Confirm 200 status for AI search bots and 403 for the scrapers you blocked.
3. Add server access log monitoring for AI User-Agents. Even a weekly grep of the access log for “OAI-SearchBot” and “ClaudeBot” and “PerplexityBot” is enough to see the trend. The volume should rise within 7 to 14 days of a clean deploy.
If you want me to do the full audit including robots.txt, llms.txt, schema, and the AI citation baseline, the AI Accessibility Audit is $300 one-time and delivers in five business days. Book a free 30-minute call if you want to scope it for your site.
FAQ
What is robots.txt for AI bots?
robots.txt is a plain-text file at the domain root that tells crawlers, including AI search crawlers like OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended, which URLs they may or may not fetch. In 2026 it has become the primary control panel for AI-citation traffic.
Where does robots.txt live?
At the domain root: https://yourdomain.com/robots.txt. It must be served as text/plain with a 200 status. If your CDN intercepts the request, the edge response wins.
Should I block GPTBot?
For a service business, no. GPTBot is the training crawler. Block OAI-SearchBot or ChatGPT-User and you vanish from ChatGPT search and live queries immediately. The most common mistake in 2026 is blocking GPTBot under the assumption it controls ChatGPT visibility. It does not.
What AI bots should I always allow?
OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-NotebookLM, DuckAssistBot, Amazonbot, and MistralAI-User. Plus Googlebot, Bingbot, and Applebot.
What AI bots should I usually block?
Bytespider, meta-externalagent, FacebookBot, and CCBot. These four are the high-volume zero-value scrapers I block by default on every client site.
Does Cloudflare override my robots.txt?
Yes if you have the AI Bot blocking toggle enabled. Cloudflare returns a 403 at the edge before robots.txt is read.
How do I see which AI bots are actually crawling my site?
Check server access logs, filter by User-Agent for the bots in question, and aggregate by frequency. Cloudflare Bot Analytics shows AI bot hits in the dashboard.
Will blocking AI bots protect my content?
It will not protect against bad actors. Robots.txt is a request, not a firewall. Well-behaved bots honor it. Bad actors ignore it entirely.
Does Google-Extended block AI Overviews?
No. Google-Extended is an opt-out token specifically for Gemini and Bard training, not for AI Overviews. Blocking Google-Extended does not affect AI Overviews.
What is the right robots.txt for a service business?
Allow Googlebot, Bingbot, Applebot, all OAI-Search and Claude-Search and Perplexity user-agents, Google-Extended, GPTBot, and ClaudeBot. Block Bytespider, meta-externalagent, FacebookBot, and CCBot.
What is the right robots.txt for ecommerce?
Same allowlist as service businesses for AI search crawlers. Plus disallow faceted-nav URLs that create thin pages, internal search results, account pages, cart and checkout.
Should publishers block training crawlers?
It is the most defensible block. Publishers sell content, so allowing GPTBot, ClaudeBot, and CCBot to train models without compensation is a real business decision. Block training, allow search.
Ready to fix your AI accessibility
My AI Accessibility Audit covers robots.txt, llms.txt, schema, and the AI citation baseline as a $300 one-time package, delivered in 5 business days. If you want the full GEO retainer that includes the audit plus monthly content refactor and citation tracking, that starts at $1,500 a month on the GEO Starter tier.
Book a free 30-min call → +91 97297 12388 WhatsApp
Frequently asked questions
What is robots.txt for AI bots?
Where does robots.txt live?
Should I block GPTBot?
What AI bots should I always allow?
What AI bots should I usually block?
Does Cloudflare override my robots.txt?
How do I see which AI bots are actually crawling my site?
Will blocking AI bots protect my content?
Does Google-Extended block AI Overviews?
What is the right robots.txt for a service business?
What is the right robots.txt for ecommerce?
Should publishers block training crawlers?
Want me to do this for you?
Book a free 30-min strategy call. I’ll review your site live and ship 3 specific fixes you can use this week. No pitch.
