AI Crawler Visit Log
Overview
AI crawlers are automated programs dispatched by AI services or search services. They read public website content and may be used for training models, answering user questions, building AI search results, or general search indexing.
The "AI Crawlers" tab in "Analytics" allows you to see which known AI crawlers have visited your public website and whether they were allowed or blocked. This article only explains how to read the logs; if you want to change which crawlers can access your site, please read "Managing AI Crawler Access to Your Website".
Where to View
- Open "Analytics".
- Click "AI Crawlers".
- Select a date range to view trends, rankings, and page details.
Regular human visitors do not appear in this tab. AI crawlers do not execute front-end JavaScript tracking for typical websites, so they are counted independently as server-side logs.
Understanding Three Crawler Purposes
- Training: May be used to help AI models learn from public content.
- Retrieval: May find and cite public content when users ask questions.
- Search: Crawlers used by general search engines, such as Googlebot.
The purpose classification is based on publicly available information about known crawlers and does not represent that the final use can be precisely determined for every visit.
Understanding Verification Status
Verified
The system compares the source IP address with the network ranges published by the operating entity, confirming that the request is credibly from that official crawler.
Suspected Impersonation
The request's name appears to be a known AI crawler, but the source IP address does not match the official range. This could be a third-party impersonating the name; do not treat it as a genuine visit from the official AI service.
Unverifiable
Some services lack public IP information for comparison, or the verification could not be completed for the current request. Unverifiable does not mean malicious; it simply means the system cannot confirm its authenticity.
Allowed and Blocked
"Allowed" in the logs means the request was permitted to continue accessing the public website. "Blocked" means it matched your blocking rules in "MarTech → AI Crawlers", and the public website responded with a denial of access.
Blocked logs are still valuable. They allow you to confirm that your rules are working and which crawlers are continuously attempting access.
How to Use Page Details
Page details list the URLs that crawlers have encountered. If an important page has not been read by known crawlers, it doesn't necessarily mean it won't appear in search results; crawl frequency is determined by each service, and it takes time, especially for newly published or less content-rich websites.
Frequently Asked Questions
Not necessarily. A visit only means the crawler has read your public website. Whether it's used for training, indexing, citation, or presented in answers is still up to each service.
Not necessarily. A newly published website, low traffic, un-scheduled crawler visits, or the crawler not being on the current recognition list can all result in the report temporarily showing no data.
First, check if the request is causing abnormal traffic. If it's just a few records, you usually don't need to take immediate action. If you notice persistent anomalies, please save the date, screenshots, and URLs, then contact customer support.