AI Assistant

The AI Assistant is an optional layer that uses a Large Language Model (LLM) to add qualitative analyses on top of the crawler’s deterministic checks. It can produce a per-page SEO assessment with concrete title/description/keyword rewrites, language-aware spelling and grammar checks, an llms.txt index, your own custom policy checks, and an AI executive summary of the whole site.

It is strictly opt-in. With no --ai-* flag, nothing changes — no environment variable is read, no AI code runs, and there is zero cost. You enable it by passing at least one --ai-* flag, typically --ai-provider, --ai-model, and (for self-hosted endpoints) --ai-endpoint.

The client is a thin wrapper over the crawler’s own HTTP stack, so there are no extra binary dependencies to install.

Key Features

1. Opt-in by design: nothing runs and no key is read unless you ask for it.
2. Multiple providers: OpenAI, Anthropic, Google Gemini, and any OpenAI-compatible endpoint.
3. Six actions: per-page SEO, typo/grammar, custom prompts, llms.txt, llms-full.txt, and an executive summary.
4. Cost control: smart page ranking, hard caps, and an --ai-dry-run preview so it never blindly hits thousands of pages.
5. Safe API-key handling: env-var by default, redacted from the saved command, never serialized to JSON or the cache.
6. Content-addressed caching: re-runs are free.
7. Advisory only: AI never affects the deterministic quality score or the --ci gate (unless you explicitly opt in).

Quick Start

# OpenAI-compatible (e.g. local vLLM) — SEO analysis of the 100 most important pages
./siteone-crawler --url=https://example.com/ \
  --ai-provider=openai-compatible --ai-endpoint=http://localhost:8000/v1 \
  --ai-model=Qwen/Qwen3-32B --ai-actions=seo

# OpenAI — SEO + llms.txt (key read from OPENAI_API_KEY by default)
./siteone-crawler --url=https://example.com/ \
  --ai-provider=openai --ai-model=gpt-5-mini --ai-actions=seo,llms-txt

# Preview cost before spending anything
./siteone-crawler --url=https://example.com/ \
  --ai-provider=anthropic --ai-model=claude-sonnet-4-6 --ai-actions=seo --ai-dry-run

Providers

The AI assistant supports four providers via --ai-provider:

Value	Use for
openai	OpenAI API (key from OPENAI_API_KEY by default).
anthropic	Anthropic API (key from ANTHROPIC_API_KEY by default).
gemini	Google Gemini API (key from GEMINI_API_KEY by default).
openai-compatible	Any OpenAI-compatible endpoint — vLLM, LiteLLM, MiniMax, LocalAI, Ollama, and other self-hosted servers. Default value.

The default provider is openai-compatible. For that provider --ai-endpoint is required (it points at your server’s base API URL, e.g. http://localhost:8000/v1). For the other providers, --ai-endpoint is an optional override.

--ai-model is required when AI is enabled. Example values from the help: MiniMax-M3, gpt-5-mini, claude-sonnet-4-6, gemini-2.5-pro.

Parameter	Description
--ai-provider=<val>	openai, anthropic, gemini, or openai-compatible. Enables the AI features. Default openai-compatible.
--ai-endpoint=<url>	Base API endpoint URL. Required for openai-compatible; optional override for the others.
--ai-model=<val>	Model name to call, e.g. MiniMax-M3, gpt-5-mini, claude-sonnet-4-6, gemini-2.5-pro.

Actions

Actions are selected with --ai-actions= as a comma-separated list. The default is seo,typos,summary — the full report set. Enabling AI without specifying actions runs per-page SEO analysis, content (typo/grammar) checks, and the executive summary, all in the HTML report. custom (needs a prompt) and llms-txt/llms-full (extra files) are opt-in.

Action	What it does	Output
seo	Per-page SEO judgement with per-factor scores and recommended title/description/keywords. Complements the deterministic SEO analyzer.	”AI SEO analysis” table (text/JSON/HTML report)
llms-txt	Curated llms.txt index of the most important pages, with AI-written names and one-line summaries grouped by section.	<dir>/<domain>.llms.txt
llms-full	llms-full.txt — the selected pages’ full markdown concatenated under an AI preamble.	<dir>/<domain>.llms-full.txt
typos	Language-aware spelling, grammar, and weak-copy detection with suggestions. Skips brand names, code, and identifiers.	”AI content issues” table
custom	Runs your own prompt (--ai-prompt-file / --ai-prompt) against each page.	”AI custom check” table
summary	AI executive summary of the whole site (see below). Synthesizes the deterministic analysis (security, accessibility, SEO, performance, infrastructure) into a prioritized list of recommendations.	”AI Insights & Recommendations” box on the HTML report Summary tab

The custom action requires a prompt — see Custom Prompts. The llms-txt / llms-full.txt files are written next to --markdown-export-dir or --offline-export-dir if set, otherwise to tmp/.

AI Executive Summary (summary)

--ai-actions=summary runs after the deterministic analysis and produces a visually styled “AI Insights & Recommendations” box at the top of the HTML report’s Summary tab, below the Website Quality Score. It evaluates five areas in parallel — security, accessibility, SEO, performance, infrastructure — each grounded in compact aggregated crawl data (never raw per-URL lists), then synthesizes one cross-area, prioritized list of up to 15 actionable recommendations (fewer for a clean site — never padded) with severity, impact, and evidence.

Cost is fixed at 6 LLM calls (5 areas + 1 synthesis) regardless of site size — even for a 100,000-URL site — because only aggregates and a few capped top-N examples are sent (each area input stays well under 15 KB).

./siteone-crawler --url=https://example.com/ \
  --ai-provider=openai --ai-model=gpt-5-mini --ai-actions=summary

Tip: the area evaluations are cheap and usually run best without thinking, but the final synthesis benefits from reasoning. Use --ai-synthesis-extra-body to enable thinking only for the synthesis call — see Thinking / Reasoning.

Page Selection & Cost Control

A site can have thousands of pages, so the AI runs only on the most important ones.

Parameter	Description
--ai-include=<regex>	Only run AI on URLs matching this regex (repeatable). Applied before ranking.
--ai-exclude=<regex>	Skip AI on URLs matching this regex (repeatable, wins over --ai-include). E.g. --ai-exclude='/press/' to skip a thousand press releases.
--ai-max-pages=<int>	Hard cap on pages sent to the LLM. Default 100. The highest-ranked (most important) pages are kept. This is the primary spend control.
--ai-dry-run	Show which pages would be analyzed, the number of LLM calls, and an estimated input-token count, then exit without any API call.

Importance ranking favors the homepage, pages linked from it, shallow click-depth, hub/navigation pages, sitemap presence, and short URL paths. Only internal HTML pages with HTTP 200 are eligible.

API Key & Security

The key is resolved with the following precedence (first match wins). Prefer environment variables so the key never appears in process arguments, shell history, or logs. The crawler redacts --ai-api-key=... in the saved command and never serializes the key into JSON output or the response cache.

1. --ai-api-key-file=<file> — read the first line of a file (safest for CI; chmod 600).
2. --ai-api-key=env:VARNAME — read the named environment variable (indirection).
3. --ai-api-key=<value> — raw value (discouraged: leaks into ps/history/logs).
4. --ai-api-key-env=<name> — read the named environment variable.
5. Default: the conventional variable for the provider — OPENAI_API_KEY, ANTHROPIC_API_KEY, or GEMINI_API_KEY.

Tuning

Parameter	Description
--ai-max-tokens=<int>	Max output tokens per request. Default 32000. Auto-mapped to max_completion_tokens for OpenAI reasoning models. Raise it further if you enable thinking/reasoning (which consumes output tokens).
--ai-use-max-completion-tokens	Force max_completion_tokens instead of max_tokens (for endpoints/models that require it). Otherwise auto-detected.
--ai-temperature=<val>	Sampling temperature. Default 0.0 (omitted automatically for OpenAI reasoning models).
--ai-max-concurrency=<int>	Maximum concurrent AI requests. Default 4.
--ai-max-reqs-per-sec=<val>	Rate limit for the LLM API (requests per second).
--ai-timeout=<int>	Per-request timeout for AI calls in seconds. Default 180. Raise it for slow reasoning models.
--ai-cache-dir=<dir>	Directory for caching AI responses (content-addressed; re-runs are free). Default tmp/ai-cache. Empty value disables caching.
--ai-language=<code>	Force content language (BCP-47, e.g. cs, de) for the typos action. Auto-detected otherwise.
--ai-seo-affects-score	Let the AI SEO assessment apply a small capped deduction to the SEO quality score. Off by default — AI is advisory and never affects the --ci gate, keeping the score deterministic and reproducible.

Thinking / Reasoning

Thinking/reasoning is controlled via the universal --ai-extra-body JSON, which is deep-merged into the request body (overriding native fields). This avoids a separate switch for every provider’s differing convention:

# vLLM / Qwen (openai-compatible) — disable thinking
--ai-extra-body='{"chat_template_kwargs":{"enable_thinking":false}}'

# OpenAI reasoning effort
--ai-extra-body='{"reasoning_effort":"minimal"}'

# Anthropic extended thinking with a token budget
--ai-extra-body='{"thinking":{"type":"enabled","budget_tokens":2048}}'

# Google Gemini — disable thinking
--ai-extra-body='{"generationConfig":{"thinkingConfig":{"thinkingBudget":0}}}'

--ai-synthesis-extra-body works exactly like --ai-extra-body, but it is applied only to the final report-summary synthesis call (the summary action). Use it to enable thinking/max reasoning just for the synthesis, while the cheap per-area evaluations run without it:

--ai-extra-body='{"chat_template_kwargs":{"enable_thinking":false}}' \
--ai-synthesis-extra-body='{"chat_template_kwargs":{"enable_thinking":true}}'

For high-volume per-page analysis, thinking is usually unnecessary and increases cost/latency. When you enable thinking, keep --ai-max-tokens generous (it defaults to 32000) so reasoning does not truncate the JSON output. Models that emit inline <think>...</think> (e.g. MiniMax M3) are handled automatically — the reasoning is stripped before parsing.

Custom Prompts

--ai-prompt-file=<file> (or inline --ai-prompt=<text>) runs your prompt against each selected page (the custom action). Reference page data with placeholders — the crawler injects each value sanitized and wrapped in an XML data-boundary tag, so a naive prompt is still injection-safe:

• {{url}}
• {{title}}
• {{meta_description}}
• {{meta_keywords}}
• {{h1}}
• {{headings}}
• {{content_markdown}}
• {{lang}}
• {{browser_diagnostics}} — available only when browser rendering is enabled; injects the captured console/JS/network/security diagnostics for the page.

The model is instructed to return a JSON array of findings {severity, label, message, location}. Put your static instructions at the top of the prompt and reference {{content_markdown}} near the bottom to maximize provider prefix-cache reuse across pages.

Example — EU advertising-compliance check (compliance.txt):

You are an EU advertising-law compliance reviewer. Flag unsubstantiated superlatives
("best", "cheapest", "#1"), health/financial claims without disclosure, and misleading
urgency. Report each issue as a finding. Page content:

{{content_markdown}}

./siteone-crawler --url=https://example.com/ \
  --ai-provider=openai --ai-model=gpt-5-mini \
  --ai-actions=custom --ai-prompt-file=compliance.txt

Prompt-injection & privacy

Crawled content is wrapped in XML data tags with angle brackets escaped, and prompts instruct the model to treat it as data. Still, only send content you are comfortable sharing with the configured provider (providers may use prompt caching).

Notes

• The AI assistant is advisory: it does not gate CI/CD builds. Only the deterministic quality gate (--ci and the --ci-* thresholds) controls the exit code. The only exception is --ai-seo-affects-score (off by default), which applies a small, capped, non-deterministic deduction to the SEO category score.
• For the full list of every flag, defaults, and allowed values, see the Command-Line Options reference.