CLI: Ready-to-use Packages

Desktop Application

This section only describes the use of the command-line interface (CLI).
If you’re more interested in the desktop app, go to the desktop app page.

Where to download packages

You can download ready-to-use packages from GitHub releases for all major platforms and architectures:

• Windows 7/8/10/11 (x64)
• macOS (x64, Intel)
• macOS (arm64, Apple Silicon, M1/M2/M3)
• Linux (x64) or WSL on Windows
• Linux (arm64)

At the top of the page you will find the latest (and therefore recommended) version. Scroll down and expand Assets.

Windows 7/8/10/11 (x64)

1. Download latest Windows release *-win-x64.zip from GitHub releases, e.g. https://github.com/janreges/siteone-crawler/releases/download/v1.0.7/siteone-crawler-v1.0.7-win-x64.zip
2. Unzip *-win-x64.zip to any folder, e.g. C:\Tools\siteone-crawler-v1.0.7-win-x64
3. Press Win+R, run terminal by cmd.exe and go to the folder with the Crawler, e.g. cd C:\Tools\siteone-crawler-v1.0.7-win-x64
4. Run crawler with basic options crawler.bat --url=https://crawler.siteone.io/

Note for Windows users

Use Cygwin-based release *-win-x64.zip only if you can’t use WSL (Windows Subsystem for Linux) (typically with Ubuntu/Debian), what is recommended.

If you really have to use the Cygwin version, do not increase Windows-default --workers=1 for higher stability.

macOS (x64, Intel)

Hint for macOS users

The optimal way is to download, unzip and run the crawler from the terminal as seen below.

If you download the package from the browser to the Downloads folder, you will probably encounter permission problems. In this case, you can move the entire folder with the Crawler via the terminal to another location, for example to the homefolder ~.

# download and unpack latest release
wget https://github.com/janreges/siteone-crawler/releases/download/v1.0.7/siteone-crawler-v1.0.7-macos-x64.tar.gz
tar -xvzf siteone-crawler-v1.0.7-macos-x64.tar.gz

# go to folder
cd siteone-crawler-v1.0.7-macos-x64

# run crawler with basic options
./crawler --url=https://crawler.siteone.io/

macOS (arm64, Apple Silicon, M1/M2/M3)

# download and unpack latest release
wget https://github.com/janreges/siteone-crawler/releases/download/v1.0.7/siteone-crawler-v1.0.7-macos-arm64.tar.gz
tar -xvzf siteone-crawler-v1.0.7-macos-arm64.tar.gz

# go to folder
cd siteone-crawler-v1.0.7-macos-arm64

# run crawler with basic options
./crawler --url=https://crawler.siteone.io/

Linux (x64) or WSL on Windows

# download and unpack latest release
wget https://github.com/janreges/siteone-crawler/releases/download/v1.0.7/siteone-crawler-v1.0.7-linux-x64.tar.gz
tar -xvzf siteone-crawler-v1.0.7-linux-x64.tar.gz

# go to folder
cd siteone-crawler-v1.0.7-linux-x64

# run crawler with basic options
./crawler --url=https://crawler.siteone.io/

Linux (arm64)

# download and unpack latest release
wget https://github.com/janreges/siteone-crawler/releases/download/v1.0.7/siteone-crawler-v1.0.7-linux-arm64.tar.gz
tar -xvzf siteone-crawler-v1.0.7-linux-arm64.tar.gz

# go to folder
cd siteone-crawler-v1.0.7-linux-arm64

# run crawler with basic options
./crawler --url=https://crawler.siteone.io/