System Requirements

Hardware Requirements

CPU

The crawler can handle even one core of any common Intel/AMD CPU of the last 10 years. ARM CPUs are also supported.

Memory (RAM)

Depending on the size of your web page, the crawler may require hundreds of MB or units of GB of memory.

You can override the default memory limit of 2048M by using the --memory-limit option.

Hint for large websites

In the case of very large websites, we recommend turning on --result-storage=file. Less memory will be used at the expense of writing/reading to disk.

Disk

The crawler stores the following files on disk during operation:

• tmp/*.[html|json|txt] - directly to this folder are saved html/json/txt reports, by default with domain and timestamp in the name. These files are usually only units of MB.
• ~/.cache/siteone-crawler/http-cache/ - cache for HTTP responses of all crawled files (an XDG-compliant location that respects $XDG_CACHE_HOME). If you also crawl all assets and your whole website has 1GB, this folder will also have ~ 1GB allocated. You can change its location with --http-cache-dir, set a time-to-live with --http-cache-ttl (default 24h), or disable the cache entirely with --no-cache. You can delete the contents of this folder at any time. However, if the crawler is started repeatedly, all content will have to be downloaded again.
• tmp/result-storage/ - storage files of internal crawler results. This folder is only used when you enable --result-storage=file (you can override its location with --result-storage-dir). The default is --result-storage=memory, which does not use disk at all. You can delete the contents of this folder at any time.

Smart directory/file structure

The directory/file structure of all temporary files is designed in such a way that it is clear and you can delete it selectively or completely as needed.

For optimal performance, we recommend using an SSD or NVME disk.

Network/Internet

The crawler needs to be able to access the website you want to crawl (if you are not crawling only developer localhost).

If you want to crawl all assets, the crawler needs to be able to download all assets of the website.

The speed of the crawler also depends on the speed and latency of your internet. By default, Crawler supports brotli/gzip compression for the most efficient data transfer.

Software Requirements

Crawler is a single native binary with zero runtime dependencies and works on common Linux (x64, arm64), macOS (x64, arm64) and Windows (x64). It runs natively on Windows — no Cygwin or WSL is required. See the ready-to-use packages for downloads, and the glibc vs. musl variants for running on older Linux distributions.

The easiest is to use ready-to-use pre-built releases, and more advanced users can use manual installation.