About SiteAuditBot

SiteAuditBot is the crawler behind Site Audit. It visits a website when one of our users asks for an audit of it, and checks things like broken links, page titles and HTTPS so the site owner gets a list of fixes.

How it behaves

• It reads robots.txt first and follows it.
• It fetches at most two pages at a time, with a pause between requests.
• It stops after 50 pages per audit and only fetches public pages; it never logs in or submits forms.
• It doesn't run JavaScript or download images, videos or other files.

Blocking it

Add this to your robots.txt:

User-agent: SiteAuditBot
Disallow: /

Questions or problems: hello@example.com.