Manage AI Crawler Access to Your Website
Overview
AI companies and AI search services read public website content using automated programs called AI crawlers. You can choose which known AI crawlers are allowed to access your site, or block them.
This is an "active setup" article; if you only want to see which AI crawlers have visited, read "AI Crawler Visit Log".
Decide What You Want to Protect or Promote
Different AI crawlers have different purposes:
- Training Crawlers: May be used to help models learn from public content.
- Retrieval Crawlers: May find and cite public content when users ask questions.
- Search Engine Crawlers: Such as Googlebot and Bingbot, primarily used for traditional search results.
Blocking training crawlers typically doesn't affect Google's general search results. However, blocking retrieval crawlers might prevent your site from appearing in answers from ChatGPT, Perplexity, Gemini, Grok, or other AI services.
Where to Configure Settings
- Open "Marketing Tech".
- Open the settings panel under "AI Crawlers".
- Toggle the block status for training, retrieval, or individual crawlers as needed.
- After saving, monitor your allow/block logs in "Analytics → AI Crawlers".
By default, no blocking rules are set, meaning all currently identified AI crawlers are allowed to access your public website.
How to Block Access
Block Entire Categories
If you don't want any known training crawlers to access your content, you can disable the "Training" category. If you disable the "Retrieval" category, you'll be reminded that this may reduce your AI search visibility.
Block Individual Crawlers
If you have concerns about a specific service, you can keep categories enabled but disable individual crawlers. For example, you might choose to block only GPTBot or Google-Extended.
Search Engine Crawlers Cannot Be Blocked Here
Googlebot, Bingbot, and Applebot do not have a block switch. This is to prevent you from accidentally making your site inaccessible to traditional search engines. If your site should not appear in search results, use the noindex setting in "Pages → SEO" instead, understanding this will impact search visibility.
How Kanorio Enforces Blocking
Kanorio uses a two-layer approach:
- It adds rules to your auto-generated Robots.txt file, instructing compliant crawlers not to access your content.
- It directly responds with a 403 Forbidden status at the edge layer for known, blocked crawlers, preventing them from retrieving content.
If you switch to a fully custom Robots.txt file, the auto-generated rules won't be appended, but the edge-layer blocking will still function.
What is LLMs.txt?
Kanorio automatically provides LLMs.txt and LLMs-full.txt. These act as a site directory and a full content summary for AI tools, helping them understand your brand and page topics.
These files do not force any AI to use or cite your website, nor do they bypass your crawler blocking settings. You can preview or switch to custom content in the "SEO & Search Engines" settings.
Frequently Asked Questions
No, it won't directly affect Google's general search crawlers. Google Search uses Googlebot, while training or extended AI uses might employ different crawlers, like Google-Extended.
No. The settings block crawlers that Kanorio has identified and can block. However, unknown programs, those disguised as regular browsers, or those that don't adhere to rules may still attempt to access public content.
Historical logs are not deleted. Check the new logs generated after applying your settings to confirm that recent requests are marked as blocked.