New features: Clusters, Indexability, Ranks per level

We today released a couple of very useful new features you should know about.

  • Clusters: All important metrics for any URL segment of your site
  • Indexability report: See what gets indexed and what not
  • Pagerank and Cheirank per level: See where your valuable content is
  • Language, charset, and content-type for every page
  • Search pages by wildcard
  • Extended URL rewriting


Ever wondered, how much internal link juice flows to your product pages and how much is staying on your product listings? Ever wanted to know how many pages inside a directory are actually indexable?

Clusters to the rescue! With clusters, you may aggregate all important metrics for any part of your website. Just set up a cluster with some simple rules like "All pages starting with /products" or "All pages without a parameter page=", and we will collect all relevant data and present it to you in a single place. If more complexity is needed, Clusters may as well be defined using regular expressions.

This is how a cluster overview may look:


Pagerank and Cheirank per Level

With the new cluster functionality, you'll be able to compare important parts of your website against each other. In order to make this even more convenient, we added PageRank and Cheirank distribution not only for clusters but also over levels. This enables you to make data-driven decisions regarding if and how to optimize your site's structure.

This is how it looks:


More Data

We now collect more data per page:

  • Charset
  • Content-Type
  • Language
  • Indexability

This data of course is also available per cluster.


More improvements

We polished parts of our UI, and added some smaller features here and there. For example you now can search through all pages using wildcards. We also allowed for naming URL rewrite rules, so you can add a label to each rewrite rule applied. Our SSL handling has become smarter and now provides fallbacks for the most common problems.

Give it a try. Just start a new crawl!