Skip to content
SiteOne Crawler Logo SiteOne Crawler

Overview

What SiteOne Crawler is and who it’s for

Section titled “What SiteOne Crawler is and who it’s for”

SiteOne Crawler is the free, open-source alternative to commercial desktop crawlers like Screaming Frog and Sitebulb — but CLI-first, fully scriptable, and built to live in your CI/CD pipeline.

It crawls your entire website, runs dozens of analyses (SEO, performance, accessibility, security, best practices and content), and turns the results into a clear interactive HTML report with a 0.0-10.0 quality score — or into text / JSON for further integration. Along the way it can export the site to an offline, browsable archive, convert pages to Markdown, generate sitemaps and e-mail the report. One dependency-free binary runs on Windows, macOS and Linux.

Terminal window
# macOS / Linux (Homebrew)
brew install janreges/tap/siteone-crawler


# Audit any site — full HTML report + 0-10 quality score in one command
siteone-crawler --url=https://your-site.com

No account, no dependencies — one native binary. The HTML report path is printed at the end of the run; add --upload to share it via a private link. Not sure where to start? Run siteone-crawler with no arguments for an interactive wizard.

Typical Use Cases

Section titled “Typical Use Cases”

The same single binary serves four very different jobs. Find yours below.

For Developers

Section titled “For Developers”

Catch what you shipped by accident — before your users do.

  • Audit the quality of your work and surface imperfections you overlooked, across SEO, performance, accessibility and more.
  • Compare the impact of your optimizations before vs. after a release.
  • Pinpoint the slowest pages and the pages that aren’t cached but should be.
  • Test how your site behaves with fonts, images, styles or JavaScript turned off.
  • Crawl a still-in-progress dev build at http://localhost:PORT/, including non-public, password-protected versions.

For DevOps & CI/CD

Section titled “For DevOps & CI/CD”

A quality gate and cache-warmer for every deploy.

  • Wire the crawler into your CI/CD pipeline as a quality gate — --ci returns exit code 10 on failure so a bad build blocks the deploy.
  • Warm up the cache after a release, so the first real visitors never wait for pages to generate.
  • Stress-test performance under load by running many parallel --workers at a higher --max-reqs-per-sec.
  • Verify your anti-DoS protections (rate-limiting and connection-limiting) actually fire.

For QA Engineers

Section titled “For QA Engineers”

A second pair of eyes that notices the details other tools miss.

  • Add a thorough crawl to your test suite to catch issues other tools don’t flag.
  • Keep historical reports and track how performance, security, accessibility and SEO evolve over time.
  • Compare the same site across environments (production vs. staging) or across releases.
  • Exercise the site under heavy load with parallel workers and a higher request rate.

For Website Owners & SEO Consultants

Section titled “For Website Owners & SEO Consultants”

Know your site’s quality — and what to fix first.

  • Review your site’s overall build quality and decide which recommendations to act on.
  • Measure server-side page-build speed, e.g. to confirm SLA targets are met.
  • Spot search-engine content flaws fast: duplicate titles/descriptions, broken heading structure and more.
  • Create a functional offline copy of the site for archival or legal purposes — viewable from a USB stick, no internet required.
  • Convert the site to Markdown for documentation or for feeding content to AI models.

You can also upload and share HTML reports for free via a secure, unguessable URL.

Rewritten in Rust

Section titled “Rewritten in Rust”

Since v2.0.0 SiteOne Crawler is a single native binary with zero runtime dependencies — no PHP, Node, Python or Docker to install alongside it — and it’s faster and lighter than the original PHP version.

How it works under the hood

Section titled “How it works under the hood”

The crawler reads the starting URL, parses its links and assets, follows the discovered pages in parallel, and stops when there’s nothing new to visit — running its analyses as it goes. For the full mechanism (link-walking, local caching and graceful CTRL+C interruption), see Deep website crawling →.

See everything it can do

Section titled “See everything it can do”

This page covers the why and the who. For the complete capability index — quality scoring, CI/CD gate, AI assistant, browser rendering, the interactive wizard and the rest of the “10 tools in one” — see the full feature list →.

Ready?

Section titled “Ready?”

Run it in 30 seconds or see a live audit report.