New Crawl Servers Added

On August 19th, 2014 we added a number of new crawl servers. We also expanded to a new datacenter. The new servers are located in France and it's another step towards having crawl locations with minimized network latencies to the sites we crawl.

If you white-listed our robot using our IP-addresses instead of verifying the robot using reverse DNS lookups, you might want to add the new IPs.

You can find all information about white-listing the robot in our bot reference page.

 

Author