Ideas and Roadmap

Recently shipped

Many items once listed here as ideas are now part of the tool (since the v2.0.0 Rust rewrite):

AI assistant — optional, opt-in LLM integration for SEO suggestions, spelling/grammar checks, llms.txt generation, and custom policy prompts.
Interactive wizard — run the binary with no arguments to pick a preset mode and configure a crawl without memorizing flags.
CI/CD quality gate — configurable thresholds with a non-zero exit code on failure, plus JUnit XML and GitHub annotations.

Ideas and longer-term thinking

My best wish is that a community of interested people will form around the tool, who would like to help determine the priorities for further development. E.g. through simple votes in Discord, or liking Features in GitHub.
I would like to make the tool as easy to use as possible, so that it can be used by people who are not programmers. At the same time, I want even the most demanding users to find everything important in it, including features like the website to markdown converter and enhanced analyzers.
I also care about the high usability of the function of exporting the website to offline form. This feature has been improved with new options for content replacement and handling unwanted code. Unfortunately, it turned out that it is practically impossible to run modern websites using JS modules in offline mode using the file:// protocol due to CORS, which does not allow JS modules with the file:// protocol in the default settings of browsers. I didn’t have much success even with attempts to inline the JS module directly into HTML. It brought its own set of other problems. I will try to solve this problem in the future, but I am not sure if it will be possible. If you have any ideas, please let me know. For modern websites built on JS frameworks with SSR and hydration, it is therefore better to generate offline export with --disable-javascript, or at least --remove-all-anchor-listeners.
Already during the design of the offline export, I thought of a very useful function - to create such a version of the export, together with the corresponding configuration of Nginx/Apache/Varnish/Traefik vhost, which could work as a legitimate static mirror of a website with URL compliance, of course without real backend functionality ( forms, etc.). For websites using GraphQL, for example, Nginx vhost would also contain Lua scripts to handle these POST requests. For some of our projects, it would be great if, through GeoDNS services with auto-failover mechanisms, we could, for example, direct visitors to a completely static version of the website on our CDN in the event of an extreme DDoS attack or unexpected outages. I imagine this more robust feature would also find use for some users in the world.
Done in v2.x: measurable quality scoring is now built in — an overall score and per-category scores (Performance, SEO, Security, Accessibility, Best Practices) on a 0.0–10.0 scale, with configurable thresholds that make the crawler exit with a non-zero code via the CI/CD quality gate (--ci). Ideal for CI/CD pipelines.
A public Docker image (for a one-command docker run in CI/CD pipelines) is still on the wish list — it is not published yet. In the meantime, the dependency-free native binary and the system package repositories (apt/dnf/zypper/apk) make it easy to containerize the crawler yourself.
Most of the problems found relate to specific pages or even specific elements on the page (too bulky inline svgs, incorrect heading structure, essential elements without aria attributes, etc.). I want to create a special variant of the offline website export in the sense of “live viewing of errors”, where it will be possible to browse the entire website in the offline version, but on specific pages there will be colored/flashing highlighted elements with a tooltip detailing the error. Other errors (e.g. headers) would be displayed in semi-transparent block somewhere in the header of the page.
To make the individual findings and tables easier to interpret, I want to add a hint to each table/section as to why that area is important and how to make improvements. The enhanced analyzers provide more detailed and potentially easier-to-interpret data.
Lot of tables in the HTML report contain aggregated statistical data. I want to add some light form of graphs to the tables in the report for faster orientation (pies, lines, bars)
Done: JavaScript / SPA rendering is available via the optional browser rendering mode (--browser), which renders each page in a real Chromium so client-side-rendered (SPA) sites without SSR are crawled with their post-render DOM.
So far, only a few of the important checks have been implemented in each of the areas. In the coming weeks, we will implement more, or consolidate and improve existing ones.
A long-standing goal has been that running the tool must never require installing any runtime or dependencies. The Rust rewrite delivers exactly that — the crawler ships as a single, dependency-free native binary for every platform, so there is nothing to install beyond downloading it (or using a package manager).
Done in v2.0.0: the crawler was completely rewritten from PHP+Swoole to Rust. The original v1.x ran on PHP with the Swoole extension, which worked great on Linux/macOS but relied on Cygwin for its Windows runtime and was not very stable there. The Rust version is a single native binary that runs natively on Windows, macOS, and Linux, is roughly 25% faster and uses about 30% less memory, and has zero runtime dependencies — the Windows-stability and dependency concerns of the old stack are gone.
For extremely large websites with hundreds of thousands or millions of URLs, crawl state still has memory-tuning knobs (--max-queue-length, --max-visited-urls, --max-url-length), and response content can be offloaded to disk with --result-storage=file. A future enhancement could add a persistent store (e.g. SQLite) to resume crawling after an unexpected interruption — partially possible today thanks to the HTTP cache.