SiteOne Crawler

How it works and examples

Desktop Application

---

Netlify.com: 📊 Audit report 🚀 Web clone

---

The audit report is limited to the first 1 000 URLs using --max-visited-urls=1000.

Command-line Interface

---

Astro.build: 📊 Audit report 🚀 Web clone
Nextjs.org: 📊 Audit report 🚀 Web clone

---

The audit reports are limited to the first 1 000 URLs using --max-visited-urls=1000.
Nextjs.org website clone was done with the --disable-javascript setting to suppress unwanted features that can't work without the backend.
Indexing of example website clones by search engines is disabled using robots.txt.

3 videos, more than a thousand words

Desktop Application

Command-line interface

HTML report

First Steps

Install

Desktop Application

For most users, a desktop application that is easy to install and use will be the most suitable.

Download the latest version of the installer from releases, or here are quick links for most common platforms:

• Windows x64 - MSI (recommended) / EXE / portable EXE
• Linux x64 - AppImage (recommended) / DEB / SNAP
• macOS arm64 (M1/2/3) - DMG, macOS x64 (Intel) - DMG
  
  Note for macOS users: As of 12/12/2023, the macOS app is already signed and officially notarized by Apple. We will also strive to publish the next versions to the Mac App Store for easier installation and updates.

Command-line Interface

Use ready-to-use packages or manual installation for you platform. System requirements are minimal.

Run the first crawl

Start with basic crawler usage, see examples and then explore the rich features and dozens of command-line options.

We believe that even after watching the videos above, you already know that using the desktop application and the command-line interface are really very simple.

---

In version 1.0.8, we also implemented a free online service that allows you to upload the HTML audit report of your website to a secure unique URL for easy sharing. In the desktop application, you only need to check one checkbox, and in the command-line version, you only need to add --upload. More information on the page Online HTML report (upload).

Discover how it can help you

See how this tool can help website owners and consultants, developers, DevOps or QA engineers.

If you are missing some functionality in the tool and would appreciate it, write to us or write a feature request right away.

Share your experience

If you find this tool useful, share it with your friends or colleagues by buttons below. We have a lot of ideas and plans for further development of this tool and we want it to be useful for developers and other users around the world.

If you use GitHub and are happy with the tool, give us a GitHub Star ⭐ for both the Desktop App and the Command-line Interface repos.

Key Features

Ease of Use

We strive to make the tool to be simple to use even for less technical users and easy to install without complex dependencies. Although the tool is intended primarily for advanced users with command-line access, it can also be used through a desktop application with a graphical interface. For the easiest way to share a report with colleagues, you can use free online audit report upload feature.

Deep Website Crawling

Finds and analyzes every single page, image, font, CSS, JS or document on your website. It is really heuristic and searches even responsive images in all forms of srcset, looks for images or fonts in CSS, even parses and looks for files in the build manifests of some popular frameworks.

Analyzer and Reporter

It will perform a lot of different analyzes of your website and provide recommendations and various details in an audit report with easy sharing.

Analyzed are SEO and OpenGraph, headings, redirects and 404, performance, accessibility, security, and many other technical aspects of your website.

💡 See the generated audit report for astro.build, netlify.com and nextjs.org (all audits limited to first 1 000 URLs).

Dev/DevOps Assistant

Finds imperfections and non-optimized parts of the website that you neglected during development, helps you warm up the cache, or test the functionality of the entire website as part of your CI/CD pipelines.

By increasing the number of --workers and the --max-reqs-per-sec, you can also stress-test or test your anti-DoS mechanisms.

Clone your website for offline use

Generates a static offline version (clone, mirror) of your website that you can store and browse offline without internet and webserver. A great tool for archival purposes or robust HA (high-availability) scenarios, where regularly maintained static versions of the website can be part of DR (disaster recovery) plans.

💡 See the real export result for docs.astro.build, netlify.com and nextjs.org. Check the URL of the pages, JS, CSS, fonts, images, documents - you will see that everything is part of export, including assets from 3rd party domains.

Swiss Army Knife

This tool has a lot of features, and we plan to implement more and more useful functionality into it that community members find useful. We want this tool to be the favorite all-purpose Swiss Army Knife of every good craftsman who builds quality websites. To make it a piece in his puzzle of favorite tools.