Audisto HTTP Header Error Checker

How to detect issues with HTTP headers

Having problems with the HTTP header can lead to issues with crawling and rendering speed of the website. It can also lead to issues with different file types / char sets and even unintended client behavior when displaying the content.

Example: Audisto HTTP Header Error Check with the HTTP-header hint reports for the current crawl

Audisto HTTP Header Error Check with the HTTP-header hint reports for the current crawl

Here is the list of all specific hints related to HTTP-header errors, that can be identified with the help of the Audisto Crawler.

Table Of Content

Hints

Charset: Charset set in HTTP Content-Type header and in document differ.

Description

Both the document and the HTTP Content-Type header specify a charset, but these are not identical. Discover all occurences of conflicting duplicate charset definitions.

Examples

HTTP header:

HTTP/1.1 200 OK
Server: Apache
Date: Thu, 17 Dec 2015 15:34:23 GMT
Content-Type: text/html; charset=UTF-8
...

Meta charset (HTML 5):

<meta charset="iso-8859-1">

Meta content-equiv (HTML 4.01):

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">

XML:

<?xml encoding="iso-8859-1" ?>
Importance

If charset definitions in the HTTP header and the document differ, the browser has to use a heuristic to guess the correct charset to display the document. This may lead to problems handling the encoding of the document and slow down the rendering time for the document.

Note: There are multiple ways to specify the charset in the document that may cause the conflict, e.g. <?xml>, <meta charset> and <meta content type>.

Operating Instruction

We suggest that you set a proper charset in the HTTP header and in the document to make it easy for web clients to render the document quickly and as expected. Make sure the defined charsets are identical and not conflicting.

Charset: Invalid charset in Content-Type HTTP header

Description

The Content-Type HTTP header specifies an invalid charset. Discover all occurences of invalid charset definitions in Content-Type HTTP headers.

Examples
HTTP/1.1 200 OK
Server: Apache
Date: Thu, 17 Dec 2015 15:34:23 GMT
Content-Type: text/html; charset=foo-bar
...
Importance

If there is no valid charset defined in the HTTP header, the browser has to use the charset specified in the document or has to fall back to detect the charset to display the document. If the charset has to be guessed, this may lead to problems handling the encoding of the document. Additionally, this may slow down the rendering time for the document.

Operating Instruction

We suggest that you set a proper charset in the HTTP header and in the document to make it easy for web clients to render the document quickly and as expected. Make sure the defined charsets are identical and not conflicting.

Charset: Not set

Description

There is no charset set, neither in the Content-Type HTTP header, nor in the document, e.g. through a <meta> tag.

Importance

If there is no charset defined in the HTTP header, the browser has to fall back to detect the charset to display the document. If the charset has to be guessed, this may lead to problems handling the encoding of the document. Additionally, this may slow down the rendering time for the document.

Operating Instruction

We suggest that you set a proper charset in the HTTP header and in the document to make it easy for web clients to render the document quickly and as expected. Make sure the defined charsets are identical and not conflicting.

Charset: Not set in Content-Type HTTP header

Description

The Content-Type HTTP header does not specify a charset. Discover all URLs on the crawled website where the HTTP Content-Type header does not specify a charset. However, there may be a charset defined in the document.

Importance

If there is no charset defined in the HTTP header, the browser has to use the charset specified in the document or has to fall back to detect the charset to display the document. If the charset has to be guessed, this may lead to problems handling the encoding of the document. Additionally, this may slow down the rendering time for the document.

Operating Instruction

We suggest that you set a proper charset in the HTTP header and in the document to make it easy for web clients to render the document quickly and as expected. Make sure the defined charsets are identical and not conflicting.

Content-Security-Policy HTTP header missing

Description

If the Content-Security-Policy HTTP header is missing, it is flagged with this hint. This report indicates that your site is not protected from Cross Site Scripting (XSS) and similar security attacks, which is provided by the Content-Security-Policy header.

Example

The simplest form of Content-Security-Policy directive is one that restricts access to your own domain:

Content-Security-Policy: default-src 'self'

The directive can also be used to restrict access to only the https version of your domain:

Content-Security-Policy: default-src https://example.com
Importance

Content Security Policy is an additional level of security that can be added to your site to further protect it from attacks, most notably from Cross Site Scripting (XSS). Content Security Policy is implemented with parameters in the Content-Security-Policy HTTP header directive. Essentially, the parameters allow you to define what gets loaded from where, or in more technical terms, which domains and which resources are allowed to load.

Operating Instruction

We suggest that you use this report to identify web pages that are missing a Content Security Policy header and then add one to each of those pages.

Cookies set without secure flag

Description

If cookies are set without using the secure flag, it is flagged with this hint. This report identifies all URLs that set cookies without a secure flag.

Example

Setting the secure flag is different on each platform. For example, on ASP.NET, you use:

<httpCookies requireSSL='true'/>

In Ruby, you set for an individual cookie as follows:

cookies[:foo, :secure => true] = "bar"
Importance

Always setting cookies to be secure provides an additional level of security for your site. Session hijacking can be particularly malicious when cookie data is stolen since it typically includes user identifiers, passwords and other critical data. So it is best to ensure that cookie exchange is always done via an SSL-secured connection. This is easy to do. Simply set the secure flag for all your cookies to on.

Operating Instruction

We suggest that you use this report to identify cookies that are not secure, and if any are not, we recommend you secure all of them using the secure flag.

Redirect: Non-ASCII Characters in Location URL

Description

When processing the location HTTP header of a redirect, the crawler detected characters that are not part of the ASCII character set. This is not allowed, since only printable ASCII characters are allowed in HTTP headers.

If the crawler stumbles upon a non-ASCII redirect URL, it will try to execute it nonetheless. This however may fail, since it involves a significant amount of guessing regarding the character encoding that was used.

URLs in redirects therefore should always be correctly escaped.

Example
Location: http://example.com/ümlaut

Correct implementation:

Location: http://example.com/%C3%BCmlaut
Importance

Not all browsers or bots may be able to cope with invalid HTTP headers, causing them to stop processing the URL, or at least not follow the desired redirect. This may affect both users and search engines.

Operating Instruction

We suggest that you always use correct URL encoding, especially in redirect location HTTP headers.

Strict-Transport-Security HTTP header has short duration

Description

If the Strict-Transport-Security HTTP header is set to a short duration, it is flagged with this hint. This report identifies if your web server is NOT properly protected by HTTP Strict Transport Security (HSTS) because a short duration means your site does not have the policy applied for a sufficient period of time.

Example

The duration of your HTTP Strict-Transport-Security (HSTS) policy is defined by the max-age parameter of the Strict-Transport-Security header. Anything less than a year is considered to be a short duration. This is because if max-age is not set sufficiently high, it will expire in the short term and leave your site vulnerable.

For example, if max-age is set to 10 minutes (i.e. 600 seconds), it is considered a short duration:

Strict-Transport-Security: max-age=600

This example sets the duration to be one year which is valid duration (i.e. not a short duration):

Strict-Transport-Security: max-age=31536000

The unit of max-age is seconds, so this line implements HSTS for one year, which is 31,536,000 seconds (i.e. 365 days x 24 hours x 60 minutes x 60 seconds). So anything less than 31536000 seconds is a short duration.

If you use preloading, then the duration must be one year. Preloading is set as follows:

Strict-Transport-Security: max-age=31536000; preload
Importance

While HTTPS has emerged as a more secure protocol than HTTP, it still has vulnerabilities. So just having an SSL certificate is no longer enough to ensure the security of your site. For example, 301 Redirects from HTTP to HTTPS are susceptible to interception by hackers, and in turn, stealing of cookies or session data, or other valuable information.

HTTP Strict Transport Security (HSTS) has emerged in the last few years as a way to add another level of security to your site. HSTS is implemented using the Strict-Transport-Security directive which provides an HTTP response header back to a web browser, or user agent, at the beginning of an interaction. This directive forces the connection to use HTTPS and ignores any calls to load resources via HTTP.

The directive has a variety of parameters that can be used, the most important of which is max-age because this indicates the duration of validity for the HSTS policy. Typically, it is set to one week or one month for testing purposes and then one year or more when put into production.

Crucially, if you want to use preloading, the max-age must be set to one year. Turning preloading on means browsers can only connect to your domain using HTTPS. However, since this is a browser function, it requires some setup: you need to submit your domain and follow the apporpriate guidelines defined here. Once your domain is on the preload list, the various browsers will only allow HTTPS connections to your site.

Operating Instruction

We suggest that you use this report to identify whether the HSTS security policy has been implemented on your site for a sufficient duration and if the duration is too short consider increasing it by setting max-age to 31,536,000 seconds, or more.

Strict-Transport-Security HTTP header is invalid

Description

If a Strict-Transport-Security HTTP header is not properly defined, it is flagged with this hint. This report identifies if your server is NOT properly protected by HTTP Strict Transport Security (HSTS).

Example

An example of an invalid HSTS header is:

Strict-Transport-Security: max-age=-1

The max-age parameter must exist and be greater than or equal to 0 for the HSTS header to be valid.

An example of a valid header is:

Strict-Transport-Security: max-age=31536000

The unit of max-age is seconds, so this line implements HSTS for one year, which is 31,536,000 seconds (i.e. 365 days x 24 hours x 60 minutes x 60 seconds).

Importance

While HTTPS has emerged as a more secure protocol than HTTP, it still has vulnerabilities. So just having an SSL certificate is no longer enough to ensure the security of your site. For example, 301 Redirects from HTTP to HTTPS are susceptible to interception by hackers, and in turn, stealing of cookies or session data, or other valuable information.

HTTP Strict Transport Security (HSTS) has emerged in the last few years as a way to add another level of security to your site. HSTS is implemented using the Strict-Transport-Security directive which provides an HTTP response header back to a web browser, or user agent, at the beginning of an interaction. This directive forces the connection to use HTTPS and ignores any calls to load resources via HTTP.

The directive has a variety of parameters that can be used, the most important of which is max-age because this indicates the duration of validity for the HSTS policy. Typically, it is set to one week or one month for testing purposes and then one year or more when put into production.

Operating Instruction

We suggest that you use this report to identify whether the HSTS security policy has been properly implemented on your site using a valid HSTS header. If not, then HSTS should be properly implemented.

Strict-Transport-Security HTTP header missing

Description

If the Strict-Transport-Security HTTP header is missing, it is flagged with this hint. This report identifies if your web server is NOT protected by HTTP Strict Transport Security (HSTS), which is implemented by the Strict-Transport-Security HTTP header.

Example

A typical Strict-Transport-Security HTTP header looks like this:

Strict-Transport-Security: max-age=31536000

The unit of max-age is seconds, so this line implements HSTS for one year, which is 31,536,000 seconds (i.e. 365 days x 24 hours x 60 minutes x 60 seconds).

Importance

While HTTPS has emerged as a more secure protocol than HTTP, it still has vulnerabilities. So just having an SSL certificate is no longer enough to ensure the security of your site. For example, 301 Redirects from HTTP to HTTPS are susceptible to interception by hackers, and in turn, stealing of cookies or session data, or other valuable information.

HTTP Strict Transport Security (HSTS) has emerged in the last few years as a way to add another level of security to your site. HSTS is implemented using the Strict-Transport-Security directive which provides an HTTP response header back to a web browser, or user agent, at the beginning of an interaction. This directive forces the connection to use HTTPS and ignores any calls to load resources via HTTP.

The directive has a variety of parameters that can be used, the most important of which is max-age because this indicates the duration of validity for the HSTS policy. Typically, it is set to one week or one month for testing purposes and then longer when put into production.

Operating Instruction

We suggest that you use this report to identify whether the HSTS security policy has been implemented on your site and, if it has not been, to take steps to implement it.

Strict-Transport-Security HTTP header sent more than once

Description

If the Strict-Transport-Security HTTP header is sent more than once, it is flagged with this hint. This report identifies multiple Strict-Transport-Security HTTP header directives which can create unexpected behavior.

Example

A typical Strict-Transport-Security header looks like this:

Strict-Transport-Security: max-age=31536000

The unit of max-age is seconds, so this line implements HSTS for one year, which is 31,536,000 seconds (i.e. 365 days x 24 hours x 60 minutes x 60 seconds).

Importance

HTTP Strict Transport Security (HSTS) has emerged in the last few years as a way to add another level of security to your site. HSTS is implemented using the Strict-Transport-Security directive which provides an HTTP response header back to a web browser, or user agent, at the beginning of an interaction. This directive forces the connection to use HTTPS and ignores any calls to load resources via HTTP.

Occasionally though, more than one Strict-Transport-Security HTTP header can be sent. This is typically the result of sloppy programming, bad communication or a misconfigured server. For example, a programmer could send an HSTS header while server administrators will inject their own HSTS header. This results in multiple HSTS headers, in which case only the first is used, so either the programmer or the server administrator will see unexpected behavior on the site.

Operating Instruction

We suggest that you use this report to identify issues that have led to multiple HSTS headers and then rectify the situation by ensuring better communication among your website team or better programming to avoid multiple HSTS headers.