For a medspa, allow all five that matter: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), and Google-Extended (Google’s AI training toggle). Blocking any of them is how clinics quietly disappear from the ChatGPT and Perplexity answers that patients now read before they ever land on your site.
I’m Mandeep Singh, founder of Sprout Sage Solutions. I audit robots.txt files on aesthetic and wellness sites every week, and the single most common self-inflicted wound I find is a plugin or a developer that “hardened security” by blocking bots it didn’t recognize. The result is a clinic that ranks fine on Google but is invisible the moment a prospective patient asks ChatGPT, “who does the best morpheus8 near me?” This article is the copy-paste fix, plus the plain-English reasoning so you know exactly what each line does before you push it live.
First, the one distinction that changes everything: training vs. live retrieval
Every AI crawler does one of two jobs, and confusing them is why owners block the wrong thing.
- Training crawlers collect pages to help build or update the model. Whether you allow them affects long-term “brand memory” — whether the AI has a general sense that your clinic exists. This is GPTBot and Google-Extended.
- Live retrieval crawlers fetch pages in real time to answer a question a user just typed. These are the ones that decide whether you get cited today, with a clickable link, inside an answer. This is OAI-SearchBot and PerplexityBot.
ClaudeBot sits on the training/general side, and Anthropic also runs a separate user-triggered fetcher.
Here’s the practical takeaway: if you only care about one thing, protect the live retrieval bots. OAI-SearchBot is what pulls your page into a ChatGPT answer when someone searches inside ChatGPT. Block it and you can be the best clinic in your city with zero presence in the answer a patient is reading. I’ve watched this exact scenario cost clinics consultations they never knew they lost, because there’s no “impression” report that tells you an AI declined to mention you.
The copy-paste allowlist for a medspa robots.txt
This goes at https://yourdomain.com/robots.txt. If a file already exists, merge these rules in rather than overwriting your existing sitemap or WordPress lines.
“`
— AI crawlers a medspa WANTS —
OpenAI training crawler (feeds ChatGPT’s general knowledge of your clinic)
User-agent: GPTBot Allow: /
OpenAI live search crawler (feeds real-time ChatGPT answers + citations)
User-agent: OAI-SearchBot Allow: /
OpenAI user-triggered fetch (when someone pastes your link into ChatGPT)
User-agent: ChatGPT-User Allow: /
Anthropic / Claude crawler
User-agent: ClaudeBot Allow: /
Anthropic user-triggered fetch
User-agent: Claude-User Allow: /
Perplexity live answer crawler (citations in Perplexity results)
User-agent: PerplexityBot Allow: /
Perplexity user-triggered fetch
User-agent: Perplexity-User Allow: /
Google’s AI training/Gemini toggle (does NOT affect normal Google ranking)
User-agent: Google-Extended Allow: /
— Keep your existing rules below —
Sitemap: https://yourdomain.com/sitemap.xml “`
A few things worth understanding rather than blindly pasting:
Allow: /and the absence of aDisallowmean the same thing to a crawler. A named user-agent block with noDisallowis a full allow. I write the explicitAllow: /anyway because it documents intent — the next person who edits this file can see you meant to let these bots in.- Google-Extended is not GoogleBot. Setting it does nothing to your normal Google Search rankings. It only governs whether Google can use your content for Gemini and AI training. For a medspa that wants maximum AI visibility, allow it.
- Order and case matter less than you’d think, but spelling is unforgiving.
GPTbot(lowercase b) is notGPTBot. A typo’d user-agent is silently ignored, which means a “block” you intended never happens — or an “allow” you intended never registers.
What you should actually block (and why it’s a short list)
Allowlisting AI search does not mean opening the floodgates. There’s a legitimate category to keep out: bulk scraper and data-broker bots that take content for their own resale and send you nothing back. The two most common I flag on clinic sites:
“`
Bulk data scrapers — no benefit to a clinic
User-agent: CCBot Disallow: /
User-agent: Bytespider Disallow: / “`
CCBot is Common Crawl. Bytespider is ByteDance’s scraper. Neither drives citations or patients. Blocking them is a reasonable, low-risk call. But notice how narrow this list is — the instinct to “block AI bots” for privacy should apply to these, not to the five that decide whether you get recommended.
The verification step almost everyone skips
Publishing the file is not the finish line. I’ve seen robots.txt look perfect while a security plugin or CDN (Cloudflare, Wordfence, a “bot fight mode”) silently 403s the exact bots you just allowed. robots.txt is a request; a firewall is an enforcement. They can disagree.
Two checks I run on every clinic:
- Fetch your own robots.txt in a browser at
yourdomain.com/robots.txtand confirm the rules render as written — no plugin has rewritten them. - Check your server logs or CDN analytics for the actual user-agent strings (
GPTBot,OAI-SearchBot,PerplexityBot). If you see them arriving with200responses, you’re genuinely open. If you see403or503, your firewall is overriding your robots.txt and the allowlist is cosmetic.
On Cloudflare specifically, the “Block AI Scrapers and Crawlers” toggle and “Bot Fight Mode” will happily block OAI-SearchBot and PerplexityBot regardless of what your robots.txt says. If AI visibility matters to you, that toggle needs to be off, with the narrow scraper blocks handled in robots.txt instead.
Allowing the crawler is step one, not the whole game
Getting the bots in the door only matters if what they find is worth citing. An open robots.txt on a thin, unstructured page still loses to a competitor whose pages are built to be quoted. Two companion moves I pair with every allowlist:
- Add an
llms.txtfile so AI systems get a clean, plain-text map of your most important pages — services, pricing approach, location, and booking. I wrote the full clinic walkthrough here: the llms.txt guide for clinics. It takes the guesswork out of what an AI reads first. - Mark up your services and FAQs with structured data so answer engines can extract facts cleanly — treatment names, provider credentials, service area. The specifics that get aesthetic clinics pulled into answers are in the schema markup that gets medspas cited.
Allow, then feed, then structure. In that order. Skipping the first step makes the other two pointless, which is exactly why I lead every AI-visibility audit with the robots.txt file.
A 10-minute action plan for this week
- Open
yourdomain.com/robots.txtand read what’s actually there. Screenshot it before you touch anything. - Merge in the allowlist block above, keeping your existing
Sitemap:line. - Add the two scraper
Disallowrules if you want them — optional, not required. - Confirm your CDN/security plugin isn’t blocking the same bots at the firewall layer.
- Re-fetch the file in a browser to confirm it saved as written.
That’s the whole job. No new tools, no subscription, no code deploy on most WordPress setups — robots.txt is editable from the file manager or an SEO plugin.
If you’d rather I audit your live robots.txt, server responses, and citation readiness for you and hand back a done-for-you file, that’s exactly what I do on a free consultation. You can also reach me directly at tel:+919729712388. I’ll tell you in the first few minutes whether you’re accidentally blocking the crawlers that decide your AI visibility — most clinics I look at are.
FAQ
Does allowing AI crawlers hurt my Google rankings?
No. GoogleBot (which handles Search ranking) is separate from Google-Extended (which handles AI training). Allowing Google-Extended, GPTBot, or PerplexityBot has no effect on where you rank in normal Google results. These are independent systems with independent user-agents.
Which single crawler matters most if I can only allow one?
OAI-SearchBot. It’s the crawler that fetches pages in real time to build ChatGPT’s search answers and citations. Blocking it is the fastest way to be absent from the AI answer a patient reads before booking. PerplexityBot is a close second for the same reason.
If I block AI crawlers, does my content stay private?
Partly, and only going forward. Blocking stops compliant crawlers from taking new content, but it does nothing about pages already collected, and non-compliant scrapers ignore robots.txt entirely. For a public-facing medspa marketing site, the privacy upside of blocking is small and the cost — losing citations and recommendations — is large. Keep private data behind a login, not behind a robots.txt rule.
How do I know the crawlers are actually getting in?
Check your server logs or CDN analytics for the user-agent strings (GPTBot, OAI-SearchBot, PerplexityBot) and confirm they return a 200 status, not 403 or 503. A perfect robots.txt means nothing if a firewall or bot-protection setting is blocking those same bots at a different layer.
