The Main Domain
Each project, monitoring, and crawl is bound to exactly one main domain. This domain cannot be changed. The starting point of a crawl must be within the main domain or one of its subdomains, but can else be choosen freely.
For example a crawl with the main domain
example.com may have the following starting points:
The following starting points are forbidden, since they are defined on another host:
If you use a subdomain as main domain, like
www.example.com instead of
the starting point must be within the subdomain or one of it's subdomains.
Include or Exclude Subdomains
For each crawl, you may decide, if the crawl should include subdomains of the domains or not.
For example, giving a main domain of
example.com and a starting point of
the following domains would be included in a crawl, if including all subdomains is enabled.
If, however, including all subdomains is disabled, only the domains of the starting point would be included:
Additional domains are available for projects only.
Sometimes it may be necessary to crawl several domains at once. This is possible using additional domains.
Think for example of a site that has been localized using different top-level-domains:
Or think of a site that has outsourced some functionality to some satellite-domains:
Additional domains must be added in the project settings first.
This feature is only available, if the main domain is verified.
To edit a project, choose Edit Settings on your project dashboard. There you may add an remove domains in the section titled Domains.
Excluded domains are on the left, included domains on the right. You can move domains from left to right and back by simply clicking on them in one of the according lists.
Only verified domains can be added as additional domains.
Every included additional domain and all of its subdomains will be included in the crawl. Given for example an
additional domain of
www.example.fr, the following domains would be crawled:
Once additional domains have been configured in the project settings, they can be enabled or disabled per crawl withing the advanced settings section.
Monitoring crawls always have additional domains enabled, if configured for the project.