Extending

SiteOne Crawler is written in Rust and built with a modular, trait-based architecture. You can extend it with your own analyzers, exporters, and content processors by adding Rust code to the project and rebuilding it.

Architecture Overview

The crawler exposes three main extension points, each defined as a Rust trait:

Extension type	Trait	Lives in	Registered in
Analyzer	`Analyzer`	`src/analysis/`	`src/engine/initiator.rs`
Exporter	`Exporter`	`src/export/`	`src/engine/manager.rs`
Content processor	`ContentProcessor`	`src/content_processor/`	`src/engine/manager.rs`

The recommended way to start is to copy an existing component of the same type, rename it, and adapt it — the existing modules are the best reference for the conventions used across the codebase.

Creating a Custom Analyzer

Analyzers examine crawled data after (and optionally during) the crawl and produce tables, summary items, and findings that flow into the text/JSON/HTML output. This is the most common extension point.

The `Analyzer` trait

Every analyzer implements the Analyzer trait defined in src/analysis/analyzer.rs:

pub trait Analyzer: Send + Sync {
    /// Post-crawl analysis. Called once after all URLs have been visited.
    /// Use `status` to read crawl results and `output` to emit tables/findings.
    fn analyze(&mut self, status: &Status, output: &mut dyn Output);

    /// Optional per-URL analysis, called for each just-visited URL while crawling.
    /// Body and headers are already downloaded and decompressed.
    /// Returns None by default (no per-URL work).
    fn analyze_visited_url(
        &mut self,
        _visited_url: &VisitedUrl,
        _body: Option<&str>,
        _headers: Option<&HashMap<String, String>>,
    ) -> Option<UrlAnalysisResult> { None }

    /// Optionally expose per-URL results as an extra table column (None by default).
    fn show_analyzed_visited_url_result_as_column(&self) -> Option<ExtraColumn> { None }

    /// Whether this analyzer should run (e.g. based on options).
    fn should_be_activated(&self) -> bool;

    /// Execution order — lower numbers run earlier.
    fn get_order(&self) -> i32;

    /// Unique analyzer name (used for filtering, stats, logging).
    fn get_name(&self) -> &str;

    /// Self-timing accessors (delegate these to the embedded BaseAnalyzer).
    fn get_exec_times(&self) -> &HashMap<String, f64>;
    fn get_exec_counts(&self) -> &HashMap<String, usize>;
}

A few notes on the actual pattern used in the codebase:

analyze() and should_be_activated(), get_order(), get_name(), get_exec_times(), get_exec_counts() are required. The per-URL methods (analyze_visited_url, show_analyzed_visited_url_result_as_column) have default implementations, so override them only if you need real-time analysis.
Each analyzer struct embeds a BaseAnalyzer (from src/analysis/base_analyzer.rs) as a field. BaseAnalyzer holds the execution-time/count maps and provides measure_exec_time(...); you simply delegate get_exec_times()/get_exec_counts() to it.
Output is produced via the Output trait — typically by building a SuperTable (see src/components/super_table.rs) and calling output.add_super_table(...), plus status helpers like add_summary_item_by_ranges(...) for summary lines.

src/analysis/external_links_analyzer.rs is a compact, self-contained example that follows exactly this shape — read it alongside this page.

Walkthrough: add an analyzer

Create the module src/analysis/my_analyzer.rs:

use std::collections::HashMap;

use crate::analysis::analyzer::Analyzer;
use crate::analysis::base_analyzer::BaseAnalyzer;
use crate::output::output::Output;
use crate::result::status::Status;

pub struct MyAnalyzer {
    base: BaseAnalyzer,
}

impl Default for MyAnalyzer {
    fn default() -> Self { Self::new() }
}

impl MyAnalyzer {
    pub fn new() -> Self {
        Self { base: BaseAnalyzer::new() }
    }
}

impl Analyzer for MyAnalyzer {
    fn analyze(&mut self, status: &Status, output: &mut dyn Output) {
        // Read crawl results from `status`, build a SuperTable,
        // call output.add_super_table(...), and add summary items.
    }

    fn should_be_activated(&self) -> bool { true }
    fn get_order(&self) -> i32 { 200 } // pick an order after the built-ins
    fn get_name(&self) -> &str { "MyAnalyzer" }

    fn get_exec_times(&self) -> &HashMap<String, f64> { self.base.get_exec_times() }
    fn get_exec_counts(&self) -> &HashMap<String, usize> { self.base.get_exec_counts() }
}

Declare the module in src/analysis/mod.rs:
```
pub mod my_analyzer;
```
Register the analyzer in src/engine/initiator.rs, inside register_analyzers(...), alongside the existing register_analyzer(...) calls:
```
use crate::analysis::my_analyzer::MyAnalyzer;
// ...
analysis_manager.register_analyzer(Box::new(MyAnalyzer::new()));
```
If your analyzer should only run when a certain option is set, follow the pattern used for option-driven analyzers (construct it, call a setter such as set_activated(...)/set_config(...), then register it). The manager then calls auto_activate_analyzers() and applies any --analyzer-filter-regex.
Rebuild:
Terminal window
```
cargo build --release
```

Your analyzer now runs on every crawl (subject to should_be_activated()), and its tables appear in the text, JSON, and HTML outputs.

Creating a Custom Exporter

Exporters turn crawl results into output — a file, an email, an upload, a sitemap, an offline clone, etc. They implement the Exporter trait in src/export/exporter.rs:

pub trait Exporter: Send + Sync {
    /// Name of the exporter (for logging/debugging).
    fn get_name(&self) -> &str;

    /// Whether this exporter runs, based on the configured options.
    fn should_be_activated(&self) -> bool;

    /// Perform the export. Report progress/results via the Output trait.
    fn export(&mut self, status: &Status, output: &dyn Output) -> CrawlerResult<()>;
}

Existing exporters live in src/export/ — file_exporter.rs, sitemap_exporter.rs, offline_website_exporter.rs, markdown_exporter.rs, mailer_exporter.rs, and upload_exporter.rs (plus the HTML report under src/export/html_report/). To add one:

Create src/export/my_exporter.rs and implement Exporter.
Declare it in src/export/mod.rs (pub mod my_exporter;).
Wire it into the crawl in src/engine/manager.rs, where the other exporters are constructed and pushed onto the exporters: Vec<Box<dyn Exporter>> list (each gated by its own option/condition).
Rebuild with cargo build --release.

Creating a Custom Content Processor

Content processors parse a content type (HTML, CSS, JS, XML, or framework-specific output) to discover URLs and to rewrite content for offline/markdown export. They implement the ContentProcessor trait in src/content_processor/content_processor.rs, whose key methods are:

find_urls(&self, content, source_url) -> Option<FoundUrls> — extract URLs from the content.
apply_content_changes_before_url_parsing(...) — mutate content prior to URL extraction.
apply_content_changes_for_offline_version(...) — rewrite content for the offline clone.
is_content_type_relevant(&self, content_type) -> bool — declare which content types this processor handles.
get_name(&self) and set_debug_mode(&mut self, ...).

Existing processors in src/content_processor/ include html_processor.rs, css_processor.rs, javascript_processor.rs, xml_processor.rs, and the framework-specific astro_processor.rs, nextjs_processor.rs, and svelte_processor.rs. To add one:

Create src/content_processor/my_processor.rs and implement ContentProcessor.
Declare it in src/content_processor/mod.rs.
Register it in src/engine/manager.rs, inside create_content_processor_manager(...), with cpm.register_processor(Box::new(MyProcessor::new(config.clone()))) — order matters, so place it relative to the existing processors as needed.
Rebuild with cargo build --release.

Optional `browser` Feature

If your extension depends on browser-rendering functionality (the chromiumoxide/CDP engine in src/browser/), remember that this code only compiles when the browser Cargo feature is enabled. The browser feature is on by default, so the default build and the pre-built binaries already include it. Still, guard any browser-only code with #[cfg(feature = "browser")] so the lean build keeps compiling:

# Default build — includes the browser feature
cargo build --release

# Lean build — drops the browser engine
cargo build --release --no-default-features

Best Practices for Extensions

Copy an existing component of the same type as your starting point — it encodes the project’s conventions.
Keep Send + Sync — all three traits require it because crawling is concurrent. Avoid non-thread-safe state.
Be memory-conscious — crawls can cover very large sites; stream and aggregate rather than hold everything in memory.
Pick a sensible get_order() for analyzers so your output appears in a logical place relative to the built-ins.
Return errors via CrawlerResult (for exporters) instead of panicking — don’t crash the whole crawl.
Run cargo fmt and cargo clippy and add tests; see Contribution and Development.

If you build something broadly useful, consider contributing it upstream so the whole community benefits.