Troubleshooting

Even though SiteOne Crawler is designed to work smoothly across various environments and websites, you may occasionally encounter issues that require troubleshooting. This guide provides solutions to common problems and guidance on debugging more complex issues.

Common Issues and Solutions

Memory Issues

Problem: Crawler crashes with “out of memory” errors when crawling large websites.

Solutions:

1. Adjust memory limit: Use the --memory-limit option to increase available memory (e.g., --memory-limit=4096M for 4GB).
2. Switch to file storage: Use --result-storage=file instead of the default memory storage for large websites.
3. Enable compression: Add --result-storage-compression to reduce memory footprint (at the cost of some CPU usage).
4. Limit crawl scope: Use --max-depth, --max-visited-urls, or regex filters to reduce the number of pages crawled.

Performance Issues

Problem: Crawling is too slow or times out on large sites.

Solutions:

1. Increase concurrency: Use --workers to increase the number of concurrent requests (default is 3, consider 5-10 for faster crawling).
2. Adjust timeout: Use --timeout to set a higher timeout value for slow-responding servers.
3. Disable unnecessary features: Use options like --disable-javascript, --disable-images, etc. if you don’t need to analyze those resources.
4. Use the HTTP cache: Make sure you’re not disabling the cache with --http-cache-dir='off' during repeated runs.

Crawling Problems

Problem: Crawler can’t access certain pages or doesn’t find all links.

Solutions:

1. Check robots.txt constraints: The crawler respects robots.txt by default. Use --ignore-robots-txt if needed for internal analysis.
2. Examine JavaScript links: For SPAs or JavaScript-heavy sites, many links might be generated dynamically and not discovered. Enable browser rendering (--browser) so the crawler renders each page in a real Chromium and sees the post-render DOM, or add key URLs manually.
3. Domain restrictions: If your site spans multiple domains or subdomains, use --allowed-domain-for-crawling to include them.
4. Authentication: For protected sites, use --http-auth for Basic authentication.

Export and Report Issues

Problem: Reports or exports are incomplete or contain errors.

Solutions:

1. Verify exported paths: Check that the output directories exist and are writable.
2. Encoding issues: For websites with non-standard character encodings, consider using --replace-content to fix encoding problems.
3. Large reports: For extensive sites, HTML reports might be too large for browsers. Consider using JSON output and filtering the data.

Debugging Techniques

Enable Debug Mode

For detailed debugging information, use the --debug flag:

./siteone-crawler --url=https://example.com/ --debug

For targeted debugging of specific URLs, use the --debug-url-regex option:

./siteone-crawler --url=https://example.com/ --debug-url-regex='/about/'

Logging to File

To capture debug output to a file without displaying it in the console:

./siteone-crawler --url=https://example.com/ --debug-log-file=debug.log

Progressive Testing

When troubleshooting complex issues:

1. Start small: Begin with a single page (--single-page) to verify basic functionality
2. Add complexity gradually: Incrementally add more pages and features to identify where issues occur
3. Isolate components: Use flags like --analyzer-filter-regex to focus on specific analyzers

Specific Scenarios

Handling Modern JavaScript Frameworks

If you’re crawling sites built with React, Vue, Angular, or other modern frameworks:

1. Enable browser rendering (--browser) for purely client-rendered (SPA) sites — it renders each page in a real Chromium so links and content that only appear after JavaScript runs are discovered
2. Use --remove-all-anchor-listeners to bypass client-side routing that might prevent normal link discovery
3. Consider manually discovering key URLs as SPA links might not be found through standard HTML parsing

Working with Large E-commerce Sites

For large e-commerce sites with thousands of product pages:

1. Use --max-depth=2 to focus on category pages rather than all products
2. Consider --result-storage=file and --result-storage-compression for memory efficiency
3. Use regex filtering to focus on important sections: --include-regex='/product/|/category/'

Handling Sites with Login Requirements

For sites requiring authentication:

1. Basic HTTP authentication: Use --http-auth=<username:password>
2. For complex login forms, consider pre-authenticating and using cookies (this requires custom development)

Error Messages Explained

Error Message	Likely Cause	Solution
GLIBC_2.xx not found	The glibc binary is too new for an older Linux distribution (e.g. Ubuntu 22.04, Debian 12)	Use the statically linked musl binary instead — it runs on any distribution. See ready-to-use packages
Crawl aborts on a very large site / high memory usage	Crawl state and response bodies kept in memory	Raise --memory-limit (e.g. --memory-limit=4096M) and/or switch to --result-storage=file to offload content to disk
Operation / request timed out	Server not responding within the timeout period	Increase the request timeout with --timeout
Connection refused / cannot reach host	Wrong URL, port, proxy, or DNS resolution	Verify the URL and port, check --proxy/--resolve, and confirm the host is reachable

Getting More Help

If you’re still experiencing issues after trying these solutions:

1. Check the GitHub repository for recent issues or discussions
2. Open a new issue with detailed information about your problem
3. Include your command with all options, environment details, and relevant error messages

💡Further Troubleshooting Tips

• When memory issues persist, consider breaking a large site into smaller segments using --include-regex
• For cross-origin resource problems, ensure all domains are properly added with --allowed-domain-for-external-files
• Regularly update SiteOne Crawler to benefit from bug fixes and performance improvements
• The crawler is a single self-contained binary with no runtime dependencies — if it fails to start on an older Linux distribution, use the statically linked musl variant (see ready-to-use packages)