Crawl Several Domains At Once

Since August 25th, 2015, we allow the crawling of several domains at once. This feature is available to all projects. To cope with the additional data when crawling across domains, we also increment the bookable capacity to 50 million crawled pages.

Crawling several domains comes in handy, if you outsourced some functionally to other domains or provide localized versions of you page with different top-level domain - think of a German page under .de, a French version under .fr, and the English one under .com. With a crawl that spans across all those domains you can detect broken links between them. In addition you can analyze the complete structure of the project and identify your most important pages by looking at the calculated authority and hub metrics.

Additional domains must be defined per project, whereas the main domain is fixed, and the starting point is limited to the main domain. Crawling these domains can be switched on and off per crawl. See our help section for details.

The number of additional domains is unlimited, however they must be verified, and so must the project's main domain.

 

Author