Browser Rendering

Browser rendering is an optional mode (--browser) that renders each page in a real Chromium browser via the Chrome DevTools Protocol (CDP) instead of issuing a plain HTTP request. This lets you crawl JavaScript / SPA sites with their post-render DOM — client-side links, hydrated content, and framework markup. Link extraction, offline export, and markdown export then all see the rendered page.

Browser rendering is off by default: with --browser off, the crawler behaves exactly as a plain HTTP client.

Built into the default build

Browser rendering is built into the default build and the pre-built binaries — it adds only the ~6 MB chromiumoxide CDP client, not a browser (the actual Chromium is detected or downloaded at runtime, never bundled). You just pass --browser; no special build is needed. For a lean binary without it, compile with cargo build --release --no-default-features. See Build from Source for details.

Key Features

1. Post-render DOM crawling: JavaScript / SPA sites are crawled as a user sees them.
2. Screenshots: a screenshot of every crawled page — viewport (custom resolution) or full-page (entire scroll height), as PNG/JPG/WebP.
3. Screenshot animation: stitch all the page screenshots into a GIF or MP4 video time-lapse of the whole crawl (MP4 needs ffmpeg; GIF doesn’t).
4. Console / error diagnostics: per-page JavaScript console errors, uncaught exceptions, failed sub-requests (404/5xx), and CSP/CORS/mixed-content violations.
5. Headless by default, or --browser-headful to watch each page open.
6. Easiest possible setup, no Node.js: auto-detects an installed Chrome/Chromium/Edge/Brave, and if none is found it offers a one-time chrome-headless-shell download — or point it at any browser with --browser-path.

Quick Start

# Crawl a JavaScript / SPA site with full browser rendering
./siteone-crawler --url=https://my.spa.tld --browser

# Capture a full-page screenshot of every page (PNG by default)
./siteone-crawler --url=https://my.spa.tld --browser --screenshots --screenshot-mode=full-page

# Screenshot every page, then stitch them into a GIF + MP4 time-lapse of the crawl
./siteone-crawler --url=https://my.spa.tld --browser --screenshots --screenshots-animation=gif,mp4

# Watch it run in a visible window (one page at a time)
./siteone-crawler --url=https://my.spa.tld --browser --browser-headful

# Use a specific browser binary and wait until the network goes idle
./siteone-crawler --url=https://my.spa.tld --browser --browser-path=/usr/bin/google-chrome --browser-wait=networkidle

Browser Detection & Download

The browser is auto-detected (Chrome, Chromium, Edge, or Brave). If none is found, you are offered a one-time chrome-headless-shell download. The auto-download trusts Google’s CDN over TLS.

• --browser-auto-download — pre-consent to downloading chrome-headless-shell when no browser is found, for non-interactive/CI runs (interactive runs prompt instead).
• --browser-path=<exe> — explicit path to a Chromium/Chrome/Edge/Brave executable; skips auto-detection and download.

Headless vs. Headful & Sandbox

The browser runs headless by default. Use --browser-headful to show a visible window and watch it open each page (concurrency is kept low so the windows are watchable).

Use --browser-no-sandbox to launch Chromium with --no-sandbox. This is often required in Docker, CI, WSL, or when running as root, but it weakens the renderer’s security isolation against untrusted pages.

Browser Options

All defaults below are transcribed from the built-in help.

Option	Default	Meaning
--browser	off	Render pages in Chromium (built into the default binaries).
--browser-path=<exe>	—	Explicit browser binary; skips detection/download.
--browser-headful	off	Visible window (default is headless; renders one page at a time).
--browser-no-sandbox	off	Add --no-sandbox (often required in Docker/CI/WSL/root; weakens isolation).
--browser-auto-download	off	Pre-consent to downloading a browser in non-interactive/CI runs.
--browser-workers=<n>	3	Concurrent rendered pages (separate from --workers; browser pages are heavier).
--browser-wait=<mode>	networkidle	Page-ready wait strategy: load, domcontentloaded, or networkidle (near-idle network).
--browser-wait-extra=<ms>	0	Extra settle delay in milliseconds after the wait condition is met.
--browser-timeout=<sec>	30	Hard navigation+render timeout per page. On timeout, whatever rendered is captured.
--browser-render-all	off	Render every URL in the browser. By default only HTML documents are rendered; assets (images/CSS/JS/fonts) are fetched via HTTP.

Screenshots

Screenshots require --browser.

Option	Default	Meaning
--screenshots	off	Capture a screenshot of every rendered page.
--screenshots-dir=<dir>	tmp/screenshots/	Output directory for screenshots.
--screenshot-mode=<m>	viewport	viewport (visible area at the set resolution) or full-page (entire scroll height).
--screenshot-viewport=<WxH>	1920x1080	Viewport size used for rendering and viewport screenshots.
--screenshot-format=<f>	png	png, jpg, or webp.
--screenshot-quality=<1-100>	80	Image quality for jpg/webp screenshots.

Screenshot animation (GIF & video)

When you capture screenshots, the crawler can assemble them — in crawl order — into a single GIF and/or MP4 video time-lapse of the whole site. Frames are streamed to disk, so memory stays flat even for thousands of pages.

# Screenshot every page, then build a GIF and an MP4 time-lapse
./siteone-crawler --url=https://example.com/ --browser --screenshots \
  --screenshots-animation=gif,mp4 --screenshots-animation-frame-duration=2

Option	Default	Meaning
--screenshots-animation=<fmt>	off	Formats to build from the screenshots: gif, mp4, or gif,mp4. Requires --browser --screenshots.
--screenshots-animation-frame-duration=<s>	2	Seconds each page is shown in the animation (0.2–10).
--screenshots-animation-width=<px>	1024	Output width in pixels; height is derived from the --screenshot-viewport aspect ratio.
--ffmpeg-path=<path>	—	Explicit ffmpeg binary (auto-detected from PATH otherwise). Required for MP4.

The animation files are written next to the screenshots — by default tmp/screenshots/animation.gif and animation.mp4.

MP4 needs ffmpeg; GIF doesn’t

GIF is produced by the built-in encoder with no external dependency. MP4 requires ffmpeg — it is auto-detected on your PATH, or you can point at it with --ffmpeg-path. If MP4 is requested but ffmpeg is unavailable, the crawler fails soft: it skips the MP4 with a warning and still produces the GIF.

Hiding cookie consent banners

Cookie consent banners are fixed overlays that would otherwise cover every screenshot. --screenshot-hide-cookie-banners injects a best-effort script before each capture that clicks “reject/accept” controls across major consent platforms (OneTrust, Cookiebot, Didomi, Usercentrics incl. shadow DOM, Quantcast, TrustArc, …), removes scroll-lock, and hides remaining consent containers — plus any fixed/sticky high-z-index overlay whose text matches cookie/consent keywords (English and Czech).

./siteone-crawler --url=https://example.com/ --browser --screenshots --screenshot-hide-cookie-banners

Option	Default	Meaning
--screenshot-hide-cookie-banners	off	Best-effort dismissal/hiding of cookie consent banners before each screenshot.
--screenshot-hide-selector=<css>	—	Comma-separated CSS selectors to hide before each screenshot, e.g. "#my-banner,.overlay".

This is best-effort — no method removes 100% of banners. For a stubborn, site-specific banner, hide your own elements with --screenshot-hide-selector.

Console, JS, Network & Security Diagnostics

When browser mode is on, the crawler captures per-page diagnostics: JavaScript console errors, uncaught exceptions, failed sub-requests (404/5xx), and CSP / CORS / mixed-content violations. These appear in a “Browser issues” table.

The diagnostics are also exposed (size-bounded) to the AI assistant via the {{browser_diagnostics}} placeholder in --ai-prompt / --ai-prompt-file (the custom AI action) — e.g. you can ask the model to triage the console/network errors.

To keep the AI payload bounded, the diagnostics are capped:

Option	Default	Meaning
--console-max-messages=<int>	100	Max console/diagnostic messages per page kept for the AI payload.
--console-msg-max-chars=<int>	200	Truncate each console/diagnostic message to this many characters in the AI payload.
--console-total-max-kb=<int>	128	Total size cap (in KB) of the per-page console diagnostics AI payload.

Limitations of Browser Mode

Browser mode behaves differently from the plain HTTP path by design:

• The browser loads sub-resources and runs page JavaScript, so domain-scope / robots rules apply to the top document only.
• --http-cache-dir does not cache rendered bodies (the browser always fetches live).
• Each rendered HTML page is fetched twice — once for status/headers, once by the browser.
• HTTP auth and cookies are not forwarded to the browser.
• --proxy and --resolve are forwarded to the browser, but --resolve is applied host-only in browser mode (Chrome’s host-resolver-rules ignore the port), so per-port overrides for the same host aren’t honored by the browser the way they are on the HTTP path.

Notes

• For the full list of every flag, defaults, and allowed values, see the Command-Line Options reference.
• To feed the captured browser diagnostics to an LLM, see the AI Assistant page and the {{browser_diagnostics}} placeholder.