CLI: Manual Installation

Desktop Application

This section only describes the use of the command-line interface (CLI).
If you’re more interested in the desktop app, go to the desktop app page.

Choose your platform

• Windows 7/8/10/11 (x64)
• macOS (x64, Intel)
• macOS (arm64, Apple Silicon, M1/M2/M3)
• Linux (x64) or WSL on Windows
• Linux (arm64)

Windows 7/8/10/11 (x64)

Use WSL if possible

If using Windows, the best choice is to use Ubuntu or Debian in WSL.

If you know and use WSL, you are a lucky person :-) Continue to Linux (x64) or WSL on Windows.

Otherwise, you can download swoole-cli-v4.8.13-cygwin-x64.zip from Swoole releases and use precompiled Cygwin-based bin/swoole-cli.exe.

c:\Tools\swoole-cli-v4.8.13-cygwin-x64\bin\swoole-cli.exe C:\Tools\siteone-crawler\src\crawler.php --url=https://crawler.siteone.io/

Reliability and limited output

Running in Cygwin is not as reliable as on Linux or macOS and Cygwin does not support STDERR with rewritable lines in the console. Therefore, the output is not as beautiful as on Linux or macOS.

macOS (x64, Intel)

If using x64 macOS, you can download swoole-cli-v4.8.13-macos-x64.tar.xz and use its precompiled swoole-cli.

# clone latest crawler version
git clone https://github.com/janreges/siteone-crawler.git
cd siteone-crawler

# remove x64 Linux binary and replace with macOS x64 binary
rm bin/swoole-cli
wget https://github.com/swoole/swoole-src/releases/download/v4.8.13/swoole-cli-v4.8.13-macos-x64.tar.xz
tar -xvf swoole-cli-v4.8.13-macos-x64.tar.xz
mv swoole-cli bin/swoole-cli

# run crawler with basic options
./crawler --url=https://crawler.siteone.io/

macOS (arm64, Apple Silicon, M1/M2/M3)

If using arm64 macOS, you can download swoole-cli-v4.8.13-macos-arm64.tar.xz and use its precompiled swoole-cli.

# clone latest crawler version
git clone https://github.com/janreges/siteone-crawler.git
cd siteone-crawler

# remove x64 binary and replace with macOS arm64 binary
rm bin/swoole-cli
wget https://github.com/swoole/swoole-src/releases/download/v4.8.13/swoole-cli-v4.8.13-macos-arm64.tar.xz
tar -xvf swoole-cli-v4.8.13-macos-arm64.tar.xz
mv swoole-cli bin/swoole-cli

# run crawler with basic options
./crawler --url=https://crawler.siteone.io/

Linux (x64) or WSL on Windows

No dependencies

Crawler does not require any external dependencies under x64 Linux. It is distributed with a binary that should work as-is on all common distributions (Ubuntu, Debian, CentOS, etc.).

The easiest installation is on most Linux (x64) distributions, because git repo contains precompiled binary bin/swoole-cli.

The same procedure applies to Linux distros (typically Ubuntu/Debian) under WSL on Windows.

# clone latest crawler version
git clone https://github.com/janreges/siteone-crawler.git
cd siteone-crawler

# run crawler with basic options
./crawler --url=https://crawler.siteone.io/

Linux (arm64)

If using arm64 Linux, you can download swoole-cli-v4.8.13-linux-arm64.tar.xz and use its precompiled swoole-cli.

# clone latest crawler version
git clone https://github.com/janreges/siteone-crawler.git
cd siteone-crawler

# remove x64 binary and replace with arm64 binary
rm bin/swoole-cli
wget https://github.com/swoole/swoole-src/releases/download/v4.8.13/swoole-cli-v4.8.13-linux-arm64.tar.xz
tar -xvf swoole-cli-v4.8.13-linux-arm64.tar.xz
mv swoole-cli bin/swoole-cli

# run crawler with basic options
./crawler --url=https://crawler.siteone.io/