Skip to content
SiteOne Crawler Logo SiteOne Crawler

Basic Usage

First steps

Section titled “First steps”

If you already have the Crawler installed, basic startup is trivial.

From the folder where you installed/extracted the Crawler, you can run the following commands.

Crawl a website and print the results to the console

Section titled “Crawl a website and print the results to the console”
Terminal window
./crawler --url=https://crawler.siteone.io/

Even this basic command will analyze the specified page and save the report in various formats to tmp/ folder (by default):

  • Directorytmp/
    • report.crawler.siteone.io.20231122-195205.html
    • report.crawler.siteone.io.20231122-195205.json
    • report.crawler.siteone.io.20231122-195205.txt

Crawl with HTML report to e-mail

Section titled “Crawl with HTML report to e-mail”
./crawler \
  --url=https://crawler.siteone.io/ \
  --mail-smtp-host=smtp.example.com \
  --mail-to=my-email@example-com

Generate offline version of the website

Section titled “Generate offline version of the website”
./crawler \
  --url=https://crawler.siteone.io/ \
  --offline-export-dir=tmp/crawler.siteone.io

After executing this command, you can open the tmp/crawler.siteone.io/index.html file in your browser and you will be able to view and browse the whole website without internet connection.

Lot of other uses

Section titled “Lot of other uses”