Offline Website Generator (clone, mirror)

Real examples

See the real generated website clone for docs.astro.build, netlify.com and nextjs.org.

💡 Check the URL of the pages, JS, CSS, fonts, images, documents - you will see that everything is part of generated clone, including assets from 3rd party domains.

🤖 Indexing of sample exports/clones by search engines is disabled using robots.txt. You don’t have to worry about duplicate content :)

Features

• will help you export the entire website to offline form, where it is possible to browse the site through local HTML files (without an HTTP server) including all documents, images, styles, scripts, fonts, etc.
• you can limit what assets you want to download and export (see --disable-all-assets and other --disable-* directives) .. for some types of websites the best result is with the --disable-javascript option.
• you can specify by --allowed-domain-for-external-files from which external domains it is possible to download assets (JS, CSS, fonts, images, documents) including * option for all domains.
• you can specify by --allowed-domain-for-crawling which other domains should be included in the crawling if there are any links pointing to them. You can enable e.g. mysite.* to export all language mutations that have a different TLD or *.mysite.tld to export all subdomains.
• you can try --disable-styles and --disable-fonts and see how well you handle accessibility and semantics
• you can use it to export your website to a static form and host it on GitHub Pages, Netlify, Vercel, etc. as a static backup and part of your disaster recovery plan or archival/legal needs
• works great with older conventional websites but also modern ones, built on frameworks like Next.js, Nuxt.js, SvelteKit, Astro, Gatsby, etc. When a JS framework is detected, the export also performs some framework-specific code modifications for optimal results. For example, most frameworks can’t handle the relative location of a project and linking assets from root /, which doesn’t work with file:// mode.
• try it for your website, and you will be very pleasantly surprised :-)
• you can host the export on your own web server (e.g. Nginx) and preserve the original URL structure with --offline-export-preserve-url-structure, or keep the original URL format entirely with --offline-export-preserve-urls.
• you can use --offline-export-remove-unwanted-code (default is 1) to remove unwanted code for offline mode - typically, JS of the analytics, social networks, cookie consent, cross origins, etc.
• you can use --offline-export-no-auto-redirect-html which disables the generation of automatic sub-folder.html with meta redirects to sub-folder/index.html.
• you can use --offline-export-preserve-url-structure to preserve the original URL path structure - e.g. /about is stored as about/index.html instead of about.html. Useful when you want to deploy the clone to a web server and keep the same URL hierarchy as the original site.
• you can use --offline-export-preserve-urls to preserve the original URL format in exported HTML/CSS/JS - same-domain links become root-relative (/path), while cross-domain links stay absolute. Useful when the exported HTML is processed by tools that need production URLs (for example RAG pipelines).
• you can use --offline-export-no-url-rewriting to disable all URL rewriting in exported HTML/CSS/JS - URLs remain exactly as in the original source. Useful for RAG indexing or other processing where the original URLs must be preserved verbatim.
• you can use --replace-content option to replace content in HTML/JS/CSS before saving to disk (with strict text & regexp support). Can be specified multiple times.
• you can use --replace-query-string to replace the default behavior where the query string is replaced by a short hash constructed from the query string in filenames. You can use simple format foo -> bar or regexp in PCRE format, e.g. '/([a-z]+)=([^&]*)(&|$)/i -> $1__$2'. Can be specified multiple times.
• you can use --ignore-store-file-error to ignore any file storing errors. The export process will continue.

Browse the export with the built-in server

You don’t have to upload the clone anywhere to view it. Once it is exported, you can browse it locally with the built-in HTTP server:

./siteone-crawler --serve-offline=./tmp/mydomain.tld

This serves the offline HTML export directory over HTTP (with a Content-Security-Policy restricting assets to the same origin); no crawling is performed. By default it listens on --serve-port 8321 and binds to --serve-bind-address 127.0.0.1 (localhost only) - use 0.0.0.0 to make it reachable from other devices on your network. See --serve-offline.

JavaScript / SPA sites

For conventional websites where only the backend generates HTML, the offline export works reliably. Some modern JS frameworks, even with SSR (server-side rendering), still modify the HTML after the page is displayed - replacing links, etc. SiteOne Crawler handles many of these situations today (e.g. path substitution in JS code, or generating URL prefixes for composite JS chunk paths).

For fully client-rendered pages, the crawler now offers an opt-in browser rendering mode (--browser) that renders each page in a real Chromium browser. The offline export then sees the post-render DOM (hydrated content, client-side links, framework markup), which significantly improves results for JS-heavy and SPA sites.