Audisto HTTP Header Error Checker

How to detect issues with HTTP headers

Having problems with the HTTP header can lead to issues with crawling and rendering speed of the website. It can also lead to issues with different file types / char sets and even unintended client behavior when displaying the content.

Example: Audisto HTTP Header Error Check with the HTTP-header hint reports for the current crawl

Audisto HTTP Header Error Check with the HTTP-header hint reports for the current crawl

Here is the list of all specific hints related to HTTP-header errors, that can be identified with the help of the Audisto Crawler.

Table Of Content

Hints

Charset: Charset set in HTTP Content-Type header and in document differ.

Description

Both the document and the HTTP Content-Type header specify a charset, but these are not identical. Discover all occurences of conflicting duplicate charset definitions on the crawled website.

Examples

HTTP header

HTTP/1.1 200 OK
Server: Apache
Date: Thu, 17 Dec 2015 15:34:23 GMT
Content-Type: text/html; charset=UTF-8
...

meta charset (HTML 5)

<meta charset="iso-8859-1">

meta content-equiv (HTML 4.01)

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">

XML

<?xml encoding="iso-8859-1" ?>
Importance

If charset definitions in the HTTP header and the document differ, the browser has to use a heuristic to guess the correct charset to display the document. This may lead to problems handling the encoding of the document and slow down the rendering time for the document.

Note: There are multiple ways to specify the charset in the document that may cause the conflict, e.g. <?xml>, <meta charset> and <meta content type>.

Operating Instruction

We suggest to set a proper charset in the HTTP header and in the document to make it easy for web clients to render the document fast and as expected. Make sure the defined charsets are identical and not conflicting.

Charset: Invalid charset in Content-Type HTTP header

Description

The Content-Type HTTP header does specify an invalid charset. Discover all occurences of invalid charset definitions in Content-Type HTTP headers on the crawled website.

Examples
HTTP/1.1 200 OK
Server: Apache
Date: Thu, 17 Dec 2015 15:34:23 GMT
Content-Type: text/html; charset=foo-bar
...
Importance

If there is no valid charset defined in the HTTP header, the browser has to use the charset specified in the document or has to fall back to detect the charset to display the document. If the charset has to be guessed, this may lead to problems handling the encoding of the document. Additionally, this may slow down the rendering time for the document.

Operating Instruction

We suggest to set a proper charset in the HTTP header and in the document to make it easy for web clients to render the document fast and as expected. Make sure the defined charsets are identical and not conflicting.

Charset: Not set

Description

There is no charset set, neither in the Content-Type HTTP header, nor in the document, e.g. through a <meta> tag.

Importance

If there is no charset defined in the HTTP header, the browser has to fall back to detect the charset to display the document. If the charset has to be guessed, this may lead to problems handling the encoding of the document. Additionally, this may slow down the rendering time for the document.

Operating Instruction

We suggest to set a proper charset in the HTTP header and in the document to make it easy for web clients to render the document fast and as expected. Make sure the defined charsets are identical and not conflicting.

Charset: Not set in Content-Type HTTP header

Description

The Content-Type HTTP header does not specify a charset. Discover all URLs on the crawled website, where the HTTP Content-Type header did not specify a charset. However, there may be a charset defined in the document.

Importance

If there is no charset defined in the HTTP header, the browser has to use the charset specified in the document or has to fall back to detect the charset to display the document. If the charset has to be guessed, this may lead to problems handling the encoding of the document. Additionally, this may slow down the rendering time for the document.

Operating Instruction

We suggest to set a proper charset in the HTTP header and in the document to make it easy for web clients to render the document fast and as expected. Make sure the defined charsets are identical and not conflicting.

Content-Security-Policy HTTP Header missing

Cookies set without secure flag

Redirect: Non-ASCII Characters in Location URL

Description

When processing the location HTTP header of a redirect, the crawler detected characters that are not part of the ASCII character set. This is illegal, since only printable ASCII characters are allowed in HTTP headers.

If the crawler stumbles upon a non-ASCII redirect URL, it will try to execute it nonetheless. This however may fail, since it involves quite some guessing regarding the used character encoding.

URLs in redirects therefore should always be correctly escaped.

Example
Location: http://example.com/ümlaut

Correct implementation:

Location: http://example.com/%C3%BCmlaut
Importance

Not all browsers or bots may be able to cope with invalid HTTP headers, causing them to stop processing the URL, or at least do not follow the desired redirect. This may affect both users and search engines.

Operating Instruction

We suggest to always use correct URL encoding, especially in redirect location HTTP headers.

Strict-Transport-Security HTTP Header has short duration

Strict-Transport-Security HTTP Header is invalid

Strict-Transport-Security HTTP Header missing

Strict-Transport-Security HTTP Header send more than once