The Cernitex crawler
Cernitex collects public records, such as prices, public registers and open government data, for its research and the reports on this site. If you have seen this crawler in your server logs, this page says who it is, how it behaves and how to ask it to stop.
How it identifies itself
cernitex-corpus/0.1 (Cernitex (Pty) Ltd; cernitex.com)
Every request carries this user agent. Some add a short note on the purpose inside the brackets. It never presents itself as a browser.
How it behaves
- It reads your robots.txt and keeps to the rules it sets for all crawlers (User-agent: *): Disallow, Allow, Crawl-delay, Request-rate and Visit-time.
- It waits between requests to each host: your Crawl-delay if you set one, and half a second if you don't.
- It stops when it's refused. A 403, a 429 or a challenge page makes it stop and back off, and it never tries to get round a bot wall.
- It only reads. It doesn't sign in, fill in forms, add to a cart or buy.
- It does not publish or sell personal information.
Where it differs
A small number of named sites are collected even though their robots.txt disallows all crawlers, each by a decision recorded in our data register. Most are open government data portals whose robots.txt blocks every crawler by default. On those sites only the named paths are read, any Crawl-delay is still kept to, and a refusal still stops the crawler.
How to ask it to stop
Email hello@cernitex.com from your domain and name the site. We stop collecting from it and confirm by reply.