HTTP Headers Analysis

The HTTP Headers Analysis feature examines all response headers across your website, providing comprehensive insights into header usage, configuration patterns, and potential optimization opportunities.

Key Features

SiteOne Crawler’s HTTP Headers Analysis offers:

1. Header Prevalence: Analysis of which headers are used across your site and how frequently
2. Value Distribution: Breakdown of different values used for each header type
3. Security Headers: Special focus on security-related headers and their configuration
4. Consistency Checking: Identification of inconsistent header usage across similar resources
5. Best Practices: Validation against header implementation best practices

Analysis Tables

The analyzer generates several detailed tables:

HTTP Headers Overview

Header                    | Occurs | Unique | Values preview                                      | Min value  | Max value
---------------------------------------------------------------------------------------------------------------------------
Accept-Ranges             | 12     | 1      | bytes                                               |            |
Cache-Control             | 66     | 2      | max-age=3600 (49) / max-age=31536000 (17)           |            |
Content-Encoding          | 55     | 2      | gzip (51) / br (4)                                  |            |
Content-Length            | 16     | -      | [ignored generic values]                            | 40 B       | 8 MB
Content-Security-Policy   | 49     | 4      | [see values below]                                  |            |
Content-Type              | 67     | 10     | [see values below]                                  |            |
Strict-Transport-Security | 63     | 1      | max-age=15552000                                    |            |
X-Content-Type-Options    | 63     | 2      | nosniff (46) / nosniff, nosniff (17)                |            |
X-Frame-Options           | 63     | 1      | SAMEORIGIN                                          |            |

HTTP Header Values

Header                    | Occurs | Value
------------------------------------------------------------------------------------------------------------------------------
Cache-Control             | 49     | max-age=3600
Cache-Control             | 17     | max-age=31536000
Content-Encoding          | 51     | gzip
Content-Encoding          | 4      | br
Content-Security-Policy   | 46     | default-src 'self' 'unsafe-inline' 'unsafe-eval' data: https://example.com
X-Content-Type-Options    | 46     | nosniff
X-Content-Type-Options    | 17     | nosniff, nosniff
X-Frame-Options           | 63     | SAMEORIGIN

Security Headers Focus

The analysis pays special attention to security-related headers, with a dedicated section:

Header                     | OK    | Notice | Warning | Critical | Recommendation
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Strict-Transport-Security  | 45    | 0      | 0       | 3        | Strict-Transport-Security header is not set. It enforces secure connections and protects against MITM attacks.
X-XSS-Protection           | 45    | 0      | 0       | 3        | X-XSS-Protection header is not set. It enables browser's built-in defenses against XSS attacks.
X-Frame-Options            | 0     | 45     | 3       | 0        | X-Frame-Options header is set to SAMEORIGIN which allows this origin to embed the resource in a frame.
X-Content-Type-Options     | 45    | 0      | 3       | 0        | X-Content-Type-Options header is not set. It stops MIME type sniffing and mitigates content type attacks.
Referrer-Policy            | 45    | 0      | 3       | 0        | Referrer-Policy header is not set. It controls referrer header sharing and enhances privacy and security.
Content-Security-Policy    | 48    | 0      | 0       | 0        |

Key HTTP Headers Analyzed

The analyzer examines numerous headers, including:

Security Headers

• Content-Security-Policy: Controls resources the browser is allowed to load
• Strict-Transport-Security (HSTS): Enforces secure connections
• X-Content-Type-Options: Prevents MIME type sniffing
• X-Frame-Options: Controls if a page can be displayed in frames
• X-XSS-Protection: Enables browser’s XSS filtering
• Referrer-Policy: Controls information shared in the Referer header
• Permissions-Policy/Feature-Policy: Restricts browser features

Caching Headers

• Cache-Control: Directives for caching mechanisms
• ETag: Content validation token
• Last-Modified: Last modification date
• Expires: Expiration date for cached content

Content Headers

• Content-Type: Media type of the resource
• Content-Encoding: Compression method used
• Content-Length: Size of the entity-body
• Accept-Ranges: Server’s acceptance of range requests

Practical Benefits

The Headers Analysis provides several practical benefits:

1. Security Enhancement: Identify missing or misconfigured security headers
2. Performance Optimization: Validate caching headers for optimal performance
3. Consistency Verification: Ensure consistent header usage across your site
4. Best Practices Adherence: Compare your headers against current web standards
5. Troubleshooting: Identify unusual or problematic header configurations

Implementation Recommendations

Based on the analysis, the crawler can help you identify opportunities for improvement:

• Security Hardening: Add missing security headers like CSP, HSTS, X-Content-Type-Options
• Caching Optimization: Implement appropriate Cache-Control strategies for different content types
• Compression Efficiency: Enable modern compression methods like Brotli where supported
• Header Standardization: Ensure consistent header usage across all resources

💡Further Development Ideas

Future enhancements to the Headers Analysis could include:

• More detailed recommendations for ideal header configurations
• Header policy validation against custom rules
• Integration with major security frameworks’ header recommendations
• Historical tracking of header changes between crawls
• Custom header analysis for specialized application needs