Search engines, AI answer engines, RSS readers, shopping catalogs, and link preview systems can discover the same public catalog facts through HTML, structured data, feeds, sitemap, robots, and llms.txt.

Crawler access does not receive special ranking content: public HTML, structured data, feeds, sitemap, robots, and llms.txt describe the same products, policies, and educational resources shown to users.

Discovery files

  • Sitemap index: /sitemap-index.xml
  • Robots policy: /robots.txt
  • AI-readable map: /llms.txt
  • Product feeds: /feeds/products.json, /feeds/products.xml, /feeds/products.csv
  • Guide feeds: /feeds/articles.xml, /feeds/articles.json
  • Security contact file: /.well-known/security.txt

Crawler expectations

  • Search and AI crawlers should receive the same visible facts as users: product names, SKU, inquiry-only purchase mode, availability, lawful-use context, returns, and contact paths.
  • OpenAI, Claude, Perplexity, Applebot, Brave, Common Crawl, Googlebot, Bingbot, Pinterestbot, RSS readers, and preview bots should be able to fetch public pages and feeds without JavaScript-only metadata.
  • IndexNow submissions should be limited to real created, updated, or deleted canonical URLs and should not include parameters, search results, or faceted traps.

No crawler-specific content

  • Do not cloak different facts to shopping crawlers, search crawlers, AI crawlers, reviewers, or users.
  • Do not create hidden AI-only text, doorway pages, keyword-stuffed prompt pages, or search-result pages for indexing.
  • Do not treat llms.txt, IndexNow, or structured data as substitutes for useful visible content.