::: Wotbox ::: Web Crawler :::
The Wotbox bot crawls the internet in order to discover new and updated pages. Our index is then generated by an algorithm which selects the most important pages on the web.
Our user agent is:
We currently do our site crawls from this IP address only:
Crawling the web to a large scale is not a trivial task. Technically it is impossible to anticipate every bit of code on the internet. So we apologise if a misinterpretation has caused a problem on your website in some way. Our bot endeavours to crawl the web in an ethical manor.
- We obey the robots.txt file. Every one of our site crawls checks this file first.
- We respect the Crawl-delay parameter in the robots.txt page, our default crawl delay is 6 seconds between page loads.
- We always use the user agent stated above.
If you don't want our bot to return to your site please put the following in your robots.txt file:
If you have any queries about out bot please use our contact us page.
The Wotbox Team