Sitemaps and robots.txt for online stores: a practical guide

Two small files decide a lot about how search engines and AI tools see your store. Get them right once and check them after every big change.

Sitemap.xml: the map

A sitemap is a file, usually at /sitemap.xml, that lists the URLs you want search engines to know about, and optionally when they changed. It helps them find new products and collections faster.

• Include products, collections, key pages and blog posts.
• Exclude cart, checkout, account, internal search and duplicate pages.
• Keep it updated automatically, which most platforms do.
• Submit it in Google Search Console and check for errors.

Robots.txt: the rules

robots.txt sits at /robots.txt and tells crawlers which paths they may or may not request. A simple store example:

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Allow: /

Sitemap: https://example.com/sitemap.xml

Important points:

• It controls crawling, not indexing. A blocked page can still appear in results if other sites link to it. To keep a page out of results, use a noindex tag instead and leave it crawlable so the tag can be read.
• Do not block the CSS and JavaScript your pages need to display.
• A rule applies by path, so test before publishing.

AI crawlers

Several AI companies run crawlers that you can allow or block by name. Common examples include GPTBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity) and Google-Extended, a control that governs whether Google uses your content for some of its AI models. Names and behaviours change, so check each company's documentation.

Example allowing them:

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

If you want to be found and cited by assistants, allow them. If you prefer to keep content out of AI training, block selectively, knowing that it can reduce visibility (GEO explained).

Mistakes that hide stores

• A "Disallow: /" left over from a staging site, blocking everything.
• Blocking product images, so they never appear in image search.
• Listing redirected or broken URLs in the sitemap.
• Forgetting to update the sitemap URL after a domain change (migration guide).
• Blocking pages that you also want to noindex, so the tag is never seen.

A quick audit

1. Open yourstore.com/robots.txt. Does it look right?
2. Open /sitemap.xml. Do the URLs load and match your canonical pages?
3. In Search Console, check the Sitemaps and Pages reports for errors.
4. Test a few important URLs with the URL inspection tool.

When to review the files

Check them whenever you change theme, platform or domain, launch a new section such as a blog, or notice pages dropping out of search. A five-minute check after each of these events prevents most of the silent problems that cost stores traffic for weeks before anyone notices.

What to do next

Review both files today, and put a reminder in your calendar to recheck after every theme change or migration, when mistakes creep in.