Sitemap

A sitemap is a file that lists the pages of a website, kept at a predictable address — usually your domain followed by /sitemap.xml. It is written for machines rather than people: an inventory, in a format crawlers can read at a glance, of every page you consider worth visiting. Alongside each address it can note when the page last changed, which tells a returning crawler what is new since it was here. Where robots.txt says who may come in, the sitemap says what there is to see.

The distinction worth holding onto is that a sitemap is an invitation, not a guarantee. It tells a crawler that a page exists and is worth its attention; it does not oblige anyone to fetch it, and it does not make a thin page valuable. Crawlers also find pages the ordinary way, by following links from one page to the next. The sitemap matters most for the pages that route does not reach — the recently published, the deeply buried, the ones nothing happens to link to.

For a business trying to be found by AI, the failure is usually one of neglect rather than absence. A sitemap generated once, when the site was built, and never regenerated since, is a catalogue of a shop that has changed its stock: it lists pages that no longer exist and omits every page added since. The crawler trusts it, wastes its visit on the dead entries, and never learns about the new work at all. Nothing looks broken to a human visitor, because no human ever opens the file.

The human translation is the index at the back of a book. It doesn't make the book any better, and no one reads it for pleasure — but without it, a reader looking for one particular thing has to leaf through every page and hope. Machines reading your site are in exactly that position, and they don't leaf patiently. Keep the index current and they find what you wrote. Let it drift and they answer from whatever they happened to trip over.