Oernoe Search
OernoeBot
OernoeBot indexes public websites for Oernoe Search. It finds new sites by following links between public pages. It follows site robots rules, honors crawl delays, excludes pages marked noindex, and stores image links and attribution rather than copying image files.
Crawl limits
New websites usually receive a single homepage visit, plus one robots.txt check. Sites crawled in more depth are limited to 500 pages each, with a limit on how often each site is visited.
Website owners
To prevent crawling, publish a robots rule for the OernoeBot user agent or mark pages noindex. Site owners can also contact Oernoe Support to request removal of indexed pages.
User-agent: OernoeBot Disallow: /