Skip to content
SiteOne Crawler Logo SiteOne Crawler Logo SiteOne Crawler

Basic Usage

First steps

If you already have the Crawler installed, basic startup is trivial.

From the folder where you installed/extracted the Crawler, you can run the following commands.

Crawl a website and print the results to the console

Terminal window
./crawler --url=https://crawler.siteone.io/

Even this basic command will analyze the specified page and save the report in various formats to tmp/ folder (by default):

  • Directorytmp/
    • report.crawler.siteone.io.20231122-195205.html
    • report.crawler.siteone.io.20231122-195205.json
    • report.crawler.siteone.io.20231122-195205.txt

Crawl with HTML report to e-mail

./crawler \
  --url=https://crawler.siteone.io/ \
  --mail-smtp-host=smtp.example.com \
  --mail-to=my-email@example-com

Generate offline version of the website

./crawler \
  --url=https://crawler.siteone.io/ \
  --offline-export-dir=tmp/crawler.siteone.io

After executing this command, you can open the tmp/crawler.siteone.io/index.html file in your browser and you will be able to view and browse the whole website without internet connection.

Lot of other uses