Crawling

Crawling is a generic term for bots that scrape and scan webpages and its links and follow these to other webpages. Olbico crawls with the olbicobot.

The olbicobot

The olbicobot crawls public information of companies. We verify this information against other publicly available information. Outdated and/or incorrect data will be updated. Doing so, our users always have an actual view of a situation.
The crawling process takes place periodic. The olbicobot doesn’t crawl product or multimedia information. The olbicobot respects all crawl rules, included robots.txt and meta tags.

If you don’t want our users to have up-to-date information about your company, you can block the olbicobot. You can add a robots.txt file to your website blocking the user agent: olbicobot. With robots.txt or robots meta tags, you can allow certain sections of your website.

If you have any questions regarding the olbicobot, please do not hesitate to contact us.