Build from Source

SiteOne Crawler is written in Rust and compiles to a single self-contained binary with zero runtime dependencies. Building from source is straightforward with Cargo, Rust’s build tool.

Most users don’t need to build from source

Pre-built binaries are available for Linux, macOS, and Windows (x64 & arm64), plus Homebrew, apt, dnf, zypper, and apk packages. Building from source is only needed if you want the latest unreleased changes, a leaner binary without browser rendering, or a custom build target. See Ready-to-Use Packages for the easy path.

Prerequisites

You need Rust 1.94 or later (the minimum supported version declared in Cargo.toml). The recommended way to install Rust is via rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

After installation, verify the toolchain:

rustc --version   # should report 1.94.0 or newer
cargo --version

Build the release binary

Clone the repository and build an optimized release binary:

git clone https://github.com/janreges/siteone-crawler.git
cd siteone-crawler

# Build the optimized release binary
cargo build --release

# Run it
./target/release/siteone-crawler --url=https://my.domain.tld

The compiled binary is located at ./target/release/siteone-crawler. Copy it anywhere on your PATH (e.g. /usr/local/bin/) to run it as siteone-crawler from any directory.

Debug builds

For development and faster compile times, use cargo build (without --release). The resulting binary at ./target/debug/siteone-crawler is unoptimized and noticeably slower at runtime — use the release build for real crawls.

Browser rendering is built into the default build

The browser-rendering mode (--browser, screenshots, and console/JS/network diagnostics) is included in the default build and in the pre-built release binaries — cargo build --release already adds the ~6 MB chromiumoxide (Chrome DevTools Protocol) client. Only the client is bundled; the actual browser is detected or downloaded at runtime, never bundled. Just pass --browser:

# The default build already includes browser rendering.
cargo build --release

# Render a JavaScript / SPA site and capture screenshots
./target/release/siteone-crawler --url=https://my.spa.tld --browser --screenshots

At runtime, the crawler auto-detects an installed Chrome/Chromium/Edge/Brave, and if none is found it offers to download a chrome-headless-shell build. You can also point it at a specific browser binary with --browser-path=<exe>. See Browser Rendering for the full set of options.

If you want a leaner binary without browser rendering (drops chromiumoxide, ~6 MB smaller), build with --no-default-features:

cargo build --release --no-default-features

Build a statically linked (musl) binary

By default, the Linux build links against glibc and requires a reasonably recent system (glibc 2.39+). If you need a binary that runs on any Linux distribution regardless of the installed glibc version — for example, older distributions or minimal container images — build a statically linked binary with the musl target:

# Install the musl toolchain (Ubuntu / Debian)
sudo apt-get install musl-tools

# Add the musl Rust target
rustup target add x86_64-unknown-linux-musl

# Build a fully static binary (no system dependencies)
cargo build --release --target x86_64-unknown-linux-musl

# Run it — works on any Linux distribution
./target/x86_64-unknown-linux-musl/release/siteone-crawler --url=https://my.domain.tld

The resulting binary has no dynamic library dependencies at all.

musl performance

The musl variant is slower than the glibc variant for CPU-intensive operations (such as offline and markdown exports) due to the musl memory allocator. Prefer the glibc build on current distributions; use musl only when compatibility requires it.

Next steps

• New to the crawler? Start with the command-line options reference.
• Want to enable JavaScript/SPA crawling and screenshots? See Browser Rendering.
• Planning to contribute code? Read the Contribution and Development guide.