Our Crawler

OrleaBot

OrleaBot visits public web pages on behalf of Orlea customers to notice when those pages change. It identifies itself with the user agent OrleaBot/1.0 (+https://orlea.com/bot; competitive monitoring).

How it behaves

  • It only fetches publicly accessible pages. It never logs in, submits forms or bypasses paywalls.
  • It honours robots.txt, including rules addressed to OrleaBot specifically.
  • Each page is fetched at most once an hour and usually far less often, however many customers track it.
  • It keeps the readable text of a page to detect changes, not images or scripts.
  • When a site turns automated visitors away, a public page may be read through a fetch service instead. The same robots.txt rules, frequency limits and public-pages-only rule apply.

Opting out

Add this to your robots.txt and OrleaBot stops fetching your pages within six hours:

User-agent: OrleaBot
Disallow: /

For anything else, write to bot@orlea.com.