Dev/DevOps assistant

• allows testing public and local projects on specific ports (e.g. http://localhost:3000/)
• allows testing password-protected websites by --http-auth parameter
• allows testing website through a proxy by --proxy parameter
• will perform a stress test
• allows you to test the protection of the infrastructure against DoS attacks by --workers and --max-reqs-per-sec parameters
• will help you warm up the application cache or the cache on the reverse proxy of the entire website

Further development ideas

The main improvements in the field of DevOps will be aimed at maximizing the possibilities of using the crawler in CI/CD pipelines.

In addition to existing options such as cache warm-up, export to JSON, the ability to generate a sitemap, or a static/archive version of the website, we want to implement options for more systematic work with analysis results. E.g. to be able to define thresholds and rules regarding quality in individual areas and when these thresholds are exceeded, the crawler ends up with an error exit-code and good explanations of the problems to solve them effectively.

This will allow the crawler to be incorporated into the testing process within CI/CD, which will prevent the deployment of a new version that exhibits major issues.

If you have suggestions to improve Dev/DevOps related features, don’t be afraid to send a feature request (to desktop application, to command-line interface) with a suggestion for improvement. We are happy to consider and implement it if it will benefit more users.