WebcrawlBot

The sitemap generator’s crawler

WebcrawlBot is Webcrawl’s crawler. It never crawls the web on its own: it only visits a site when a user asks to generate its sitemap or analyse its public files (robots.txt, sitemaps, llms.txt…).

Identification

Good behaviour

• It reads your robots.txt before crawling and always obeys rules that name it.
• It downloads at most 6 pages at a time and stops at 50,000 pages per crawl.
• It submits no forms, logs into no accounts and keeps no pages.

Blocking WebcrawlBot

Add these lines to your robots.txt:

Any question: Contact page.