Oernoe Search
NewsBot
NewsBot finds new news stories and blog posts for Oernoe Search. It follows a site's public RSS or Atom feed, reads each new story once, and keeps its title, a short summary, the author and date, and links to its pictures. It works alongside OernoeBot, which indexes the rest of a site.
How it behaves
- It identifies itself as
OernoeNewsBotand reads your robots.txt first. If your robots.txt blocks OernoeNewsBot, or OernoeBot, it does not read your pages. - It honors crawl delays, waits between requests to a site, and reads only a few stories from a site at a time. It uses a conditional request for feeds, so an unchanged feed costs one small request.
- It indexes only recent posts (by default those published in the last 30 days), skips pages marked
noindex, and never fetches private-network addresses. - It stores a short summary, not the full article. Pictures are linked from your site, not copied, and open on your site.
- Stories are removed from the index after a retention period (60 days by default).
Website owners
To stop NewsBot, publish a robots rule for its user agent, or mark pages noindex. To have your news or blog feed read, make sure it is public and linked from your home page with <link rel="alternate" type="application/rss+xml">. You can also contact Oernoe Support to request removal of indexed stories.
User-agent: OernoeNewsBot Disallow: /