Kanorio

Sitemap and Search Engine Indexing

About 10 min

Overview

A Sitemap is a list of your site's pages for search engines, not a navigation page for visitors. It lists publicly accessible and indexable pages, helping search engines like Google discover your content more efficiently.

"Indexing" is the process by which search engines add pages to their database. A page must be indexed before it can appear in search results; however, being indexed doesn't guarantee a high ranking.

Kanorio's Auto-Generated Sitemap

Kanorio automatically creates a Sitemap for your published website. It includes:

  • Published and visible general pages.
  • Pages not set to noindex.
  • Published, public blog-related pages.

You can usually find it at:

yourdomain.com/sitemap.xml

Replace yourdomain.com with your actual public website URL. The Sitemap is updated whenever you publish content; you don't need to add pages manually.

What is Robots.txt

Robots.txt is a text file in your website's root directory that contains rules for compliant crawlers, specifying which content they can read. It's like a signpost at the entrance, not a password lock:

  • It helps well-behaved crawlers understand your site's rules.
  • It cannot protect publicly accessible confidential data.
  • Never rely on robots.txt alone to hide private information; sensitive data should not be placed on public pages.

Kanorio generates a default Robots.txt suitable for public websites, including the Sitemap location.

Indexing Takes Time

After publishing, confirm the following:

  1. Use an incognito window to verify your public URL loads correctly.
  2. Check in "Pages → SEO" to ensure the page is not set to noindex.
  3. Connect to Google Search Console from "SEO & Search Engines".
  4. Submit or verify your Sitemap via GSC.
  5. Wait for Google to crawl and index.

New websites or pages typically take days to weeks. If your content isn't indexed, first check for thin or duplicate content, noindex settings, or if the public website itself is inaccessible.

Customizing Sitemap and Robots.txt

Advanced options for customizing your Sitemap and Robots.txt are available under "Marketing Tech → SEO & Search Engines". Use these only if you fully understand what you are overriding:

  • A custom Sitemap replaces Kanorio's auto-generated version; you must maintain all listed URLs yourself.
  • A custom Robots.txt replaces the automatic rules; the AI crawler firewall's Edge blocking will still function, but the automatic blocking directives within Robots.txt won't be appended.

Most websites should keep the "Automatic" setting to avoid search engines misinterpreting your site due to formatting errors or missing pages.

Structured Data

Structured data is markup added to your website that helps search engines understand the type of content more accurately. Kanorio automatically generates various Schema.org data based on your site and modules, such as:

  • Organization, WebSite, and applicable LocalBusiness for the entire site.
  • FAQPage for the FAQ module.
  • VideoObject for the YouTube module.
  • Product and Offer for stores.
  • BlogPosting, Blog, and category/tag pages for blogs.
  • BreadcrumbList for navigation.

You can also add custom JSON-LD in the advanced settings under "SEO & Search Engines". This is for advanced users familiar with structured data formats; if you're unsure, keep the automatic settings to prevent adding incorrect or duplicate markup.

Frequently Asked Questions

Search engines decide when to recrawl and re-index your pages. First, ensure the page is published, loads correctly, is not set to noindex, and that you've submitted your Sitemap via GSC.

In most cases, no. The auto-generated Sitemap updates with your published, visible pages; a custom version requires ongoing manual maintenance.

No. It's a directive for crawlers, not an access control mechanism. Do not place sensitive information on any public page.