Audisto Crawler Hints
Robots Directive Analyzer Crawl Control & Indexation Tool
What is the robots directive checker for
Control Search Engine Access: Detect issues with robots directives including conflicting specifications, duplicate directives, and implementation errors that affect crawling.
Ensure Proper Indexation: Find problems with noindex/nofollow implementations, CSP blocking, and directive conflicts that impact search visibility.
Maintain Crawl Efficiency: Identify improper or missing robots directives that could waste crawl budget or prevent important content from being indexed.
Monitor Directive Health: Track changes in robots directive implementation to maintain consistent crawl control and prevent indexation issues.
How to check robots directives issues at scale
Control your indexation
Proper robots directive implementation is crucial for managing search engine access and visibility. Our analysis helps you maintain effective crawl control across your website. While our quick check provides immediate insights for single pages, our full crawler ensures consistent robots directive implementation across your entire website.
Critical Robots related hints
Rendering: Content blocked by Content Security Policy
This hint detects all HTML documents where content rendering is blocked by a Content Security Policy during rendering.Learn more
Rendering: Inline content blocked by Content Security Policy
This hint detects all HTML documents where inline content is blocked by the Content Security Policy during rendering.Learn more
Robots: Specified more than once
Robots directives are specified more than once. If a document contains more than one robots definition, it will be flagged.Learn more
Robots: nofollow differs across specifications
There is more than one source for robots, either a robots meta tag or a X-Robots-Tag header, and at least one specifies "nofollow" while another does not.Learn moreRobots: noindex differs across specifications
There is more than one source for robots directives, either a robots meta tag or a X-Robots-Tag header, and at least one specifies "noindex" while another does not.Learn more
Problem indicating Robots related hints
Robots: Directive probably misspelled
A directive which is likely a misspelling of the robots meta tag was found.Learn more
Informational Robots related hints
Robots: Directives Missing
No robots meta tag or X-Robots-Tag header directive was found.Learn moreRobots: follow
The URL is set to "follow" by a robots directive, either by the robots meta tag or the X-Robots-Tag header.Learn moreRobots: index
The URL is set to "index" by all robots directives, either by the robots meta tag or the X-Robots-Tag header.Learn moreRobots: nofollow
The webpage is set to "nofollow" by a robots directive, either by the robots meta tag or the X-Robots-Tag header.Learn moreRobots: noindex
The URL is set to "noindex" by a robots directive, either with the robots meta tag or the X-Robots-Tag header.Learn more