Key Features

One tool, ten jobs. Most teams stitch together a crawler, a Lighthouse-style auditor, a site-mirroring tool, an HTML-to-Markdown script, a sitemap generator and a load-tester. SiteOne Crawler does all of it — free, open source, in a single dependency-free binary.

It’s a single native binary (Windows, macOS, Linux), rewritten in Rust (v2.0.0) with zero runtime dependencies — no Node, no Python, no Docker — running ~25% faster and using ~30% less memory than the previous PHP version, with identical output.

See the most-used tool first → a live audit report, or run it in 30 seconds.

Headline capabilities

The features below are why people switch — the things competitors charge for, or that you’d otherwise script by hand.

Quality Scoring - an automatic 0.0–10.0 quality score across 5 weighted categories (Performance, SEO, Security, Accessibility, Best Practices), so you can see at a glance how well your website is optimized.
CI/CD Integration - a built-in quality gate (--ci) with configurable thresholds, JUnit XML and GitHub annotations; exit code 10 on failure lets you block bad deployments automatically (also handy as a post-deploy cache warmer).
Audit Report - the main output is one interactive HTML report that bundles every analysis and its improvement recommendations, working well on desktop and mobile.
Offline Website Generator (cloner) - generate a static clone of your site you can browse offline without a web server — ideal for archival, high-availability and disaster-recovery scenarios.
Website to Markdown Converter - export an entire website to clean Markdown to feed LLMs and documentation, with llms.txt generation and a built-in HTTP server that renders the export as styled HTML.
AI Assistant — bring your own LLM - opt-in LLM integration where you choose the model: commercial (OpenAI, Anthropic, Google Gemini) or fully local & private (vLLM, Ollama, LM Studio, or any OpenAI-compatible endpoint). It grades SEO, produces an executive summary across security, accessibility, performance and infrastructure, checks spelling/grammar, generates llms.txt, and runs your own custom prompts — with strict cost controls and a --ai-dry-run preview.
Browser Rendering - optional --browser mode renders each page in a real Chromium (CDP) so JavaScript/SPA sites are crawled, exported and converted with their post-render DOM. Run it headful (--browser-headful) to watch the browser walk through every page live, capture a screenshot of every crawled page (viewport or full-page) and stitch them into a GIF or MP4 video time-lapse (MP4 needs ffmpeg), and collect console/JS/network diagnostics.

Crawl & Analyze

Find every page and asset, then run the full analyzer suite over what it discovers.

Deep Website Crawling - finds and analyzes every single page, image, font, CSS, JS or document on your website. It is really heuristic and searches even responsive images in all forms of srcset, looks for images or fonts in CSS, even parses and looks for files in the manifests of some popular frameworks
SEO and OpenGraph Analysis - analyzes the SEO and OpenGraph settings of your website, including heading structure and the uniqueness of your titles/descriptions.
Heading Analysis - analyzes the correct semantic structure of the headings on every single page and points out errors (multiple headings <h1> or skipped heading levels are the most common)
Accessibility Analysis - analyzes and recommends at least basic accessibility rules that will take developers a few tens of minutes to implement, but will help people with disabilities work well with your website
Security Analysis - analyzes compliance with basic security recommendations, currently in the area of setting HTTP headers and cookies
Best Practices Analysis - analyzes and recommends various best practices that increase user comfort and reduce error rates (clickable phone numbers, using modern WebP/AVIF images, Brotli compression, appropriate work with SVG, using quotation marks in the HTML attributes, etc.)
Redirect and 404 Analysis - finds all URLs that are redirects or point to a non-existent page (404) so you can quickly find and fix them
Content Type Analysis - provides a detailed summary of the number and volume of individual types of content on your entire website (HTML, CSS, JS, images, fonts, documents, etc.)
Technical Analysis - will provide information and recommendations to improve your DNS and TLS/SSL configuration

Export & Convert

Turn a crawl into the artifact you actually need — a report, a static mirror, Markdown, or a sitemap.

Audit Report Sharing (Upload) - for the easiest possible sharing of the audit report, a free service is available for online sharing of the report in the form of a secure URL with optionally limited validity and a password
Exports and Reports - you can export all data to HTML report that works well on desktop and mobile, TXT (console output) or JSON for your own processing
Sitemap Generator - you can easily generate sitemap.xml and sitemap.txt for your website
Mailer - crawler can send you a detailed HTML report via your SMTP server directly to your e-mail after completing the crawling (useful, e.g. in CI/CD pipelines)

Performance & Ops

Profile what’s slow, warm caches, and catch what you shipped by accident.

Performance Analysis - crawler will provide detailed statistics on the duration of generation of individual pages, including lists of the fastest and slowest URLs, and help to find the most vulnerable parts of your website in the event of a DoS/DDoS attack.
Stress Testing - tests the performance of your website under heavy load, because requests can be made in parallel, with any number of workers (but be careful not to cause a DoS attack)
Dev/DevOps Assistant - finds imperfections and non-optimized parts of the website that you neglected during development, helps you warm up the cache, or test the functionality of the entire website as part of your CI/CD pipelines

Runs anywhere

Built to be easy to install and run for everyone, on every platform.

Interactive Wizard - run the binary without any arguments and a wizard guides you through configuration with 10 preset modes (Quick Audit, SEO Analysis, Performance Test, Security Check, Offline Clone, Markdown Export, Stress Test, and more) - no need to remember CLI flags
Availability - maximum availability for a wide range of users and platforms is our priority, so our tool is for all platforms (Windows, macOS, Linux) and both its desktop and command-line variant
Ease of Use - we strive to make the tool to be simple to use even for less technical users and easy to install without complex dependencies

How it compares

A fair, factual look at where SiteOne Crawler fits next to the popular paid desktop crawlers and a do-it-yourself toolchain. Where we can’t verify a competitor’s behavior, the cell is left as a neutral ”—”.

Capability	SiteOne Crawler	Screaming Frog	Sitebulb	DIY scripts (wget/Lighthouse/…)
Deep crawl	Yes	Yes	Yes	Partial (per script)
Quality score (0–10)	Yes	—	—	No (manual)
Offline clone / mirror	Yes	—	—	Yes (wget)
Website → Markdown	Yes	—	—	Yes (custom script)
CI/CD quality gate	Yes (`--ci`, exit code 10)	—	—	DIY
Scriptable CLI	Yes	—	—	Yes
Price	Free	Freemium (free ≤500 URLs)	Paid	Free (your time)
License	MIT (open source)	Proprietary	Proprietary	Varies
Runtime dependencies	None (single binary)	GUI app	GUI app	Node / Python / wget / …

Get started

See the most-used tool first → a live audit report, or run it in 30 seconds.