Skip to content
SiteOne Crawler Logo SiteOne Crawler

Interactive wizard

Run siteone-crawler without any arguments and an interactive wizard guides you through the configuration. You choose a crawl mode, enter the target URL, fine-tune settings with the arrow keys, and the crawler starts immediately — no need to remember CLI flags.

Terminal window
./siteone-crawler

Choosing a crawl mode

Section titled “Choosing a crawl mode”

The wizard opens with a searchable menu of 10 preset modes plus two entries for serving previously exported sites:

? Choose a crawl mode:
❯ Quick Audit               Fast site health overview — crawls all pages and assets
  SEO Analysis               Extract titles, descriptions, keywords, and OpenGraph tags
  Performance Test           Measure response times with cache disabled — find bottlenecks
  Security Check             Check SSL/TLS, security headers, and redirects site-wide
  Offline Clone              Download entire website with all assets for offline browsing
  Markdown Export            Convert pages to Markdown for AI models or documentation
  Stress Test                High-concurrency load test with cache-busting random params
  Single Page                Deep analysis of a single URL — SEO, security, performance
  Large Site Crawl           High-throughput HTML-only crawl for large sites (100k+ pages)
  Custom                     Start from defaults and configure every option manually
  ──────────────────────────────────────
  Browse offline export      Serve a previously exported offline site via HTTP
  Browse markdown export     Serve a previously exported markdown site via HTTP
[↑↓ to move, enter to select, type to filter]

Preset modes

Section titled “Preset modes”
ModeWhat it does
Quick AuditFast site health overview — crawls all pages and assets.
SEO AnalysisExtract titles, descriptions, keywords, and OpenGraph tags.
Performance TestMeasure response times with cache disabled — find bottlenecks.
Security CheckCheck SSL/TLS, security headers, and redirects site-wide.
Offline CloneDownload entire website with all assets for offline browsing.
Markdown ExportConvert pages to Markdown for AI models or documentation.
Stress TestHigh-concurrency load test with cache-busting random params.
Single PageDeep analysis of a single URL — SEO, security, performance.
Large Site CrawlHigh-throughput HTML-only crawl for large sites (100k+ pages).
CustomStart from defaults and configure every option manually.

Serve a previous export

Section titled “Serve a previous export”
EntryWhat it does
Browse offline exportServe a previously exported offline site via HTTP.
Browse markdown exportServe a previously exported markdown site via HTTP.

Settings form and equivalent CLI command

Section titled “Settings form and equivalent CLI command”

After you select a preset and enter the URL, the wizard shows a settings form where you can adjust options such as workers, timeout, content types, export options, and more.

Before the crawl starts, the wizard displays a configuration summary with the equivalent CLI command. Copy that command to reuse the exact same run later without the wizard — for example in a script or a CI pipeline.

Serving existing exports

Section titled “Serving existing exports”

If existing offline or markdown exports are detected in ./tmp/, the wizard also offers to serve them via the built-in HTTP server directly from the menu, so you can browse a previous export without re-crawling.

Section titled “Related”