Welcome, Product Hunt! Limen crosses its own threshold today. Your access code to everything premium is waiting in the launch thread.

Get your code
limen
Sign inCheck your site
I have watched these visitors arrive for years. They knock, they read, they leave — and what they carry away becomes the answer someone gets about you tomorrow. They cannot ring the bell or ask a question. Whatever your page says when they call is your whole side of the conversation.
— Janus, at the door
Glossary
Glossary

AI Crawler

An AI crawler is a piece of software sent out by a search or AI company to read web pages automatically. It announces a name when it arrives — GPTBot belongs to OpenAI, ClaudeBot to Anthropic, PerplexityBot to Perplexity, Google-Extended to Google's AI products — and it works through the web page by page, storing what it finds so the system behind it can answer questions later. It is the reason an assistant can tell anyone about your business at all.

It is worth being precise about what a crawler is not. It is not a person, and it is not browsing. It does not scroll, it does not wait for animations, it rarely clicks, and it will not hunt for anything hard to reach. It requests a page, takes whatever text comes back, and moves on. Everything an AI assistant will ever know about you from your own site is whatever survived that one plain reading.

Crawlers can be turned away, and usually are by accident. A line in robots.txt, a firewall rule, a security setting on a hosting plan, a plugin installed for an unrelated reason — any of these can refuse the visit. Because the refusal is silent and invisible in an ordinary browser, a site can sit for years perfectly readable to every human being and entirely unknown to the machines. Nothing else you do matters until the visit is allowed.

The other failure is subtler: the crawler is let in and finds nothing worth carrying away. Text locked inside images or a PDF, a page that assembles itself only after code runs, a homepage of adjectives that never states plainly what the business does or where it does it. The door was open. There was simply nothing legible on the other side.

The human translation is a visitor who calls once, cannot ask questions, and afterwards repeats whatever they managed to read. You do not get to explain, correct, or charm them. You get one plain page, read quickly — and everything anyone hears about you second-hand comes from that.