This guide walks you through configuring and initiating the Web Crawler to capture and audit your site's structure, content, and views.
nevamcx.com) and click Configure.
Customize how the crawler interacts with your pages, handles dynamic elements, and manages network security bypasses.
Crawl Mobile View: Enable this toggle to render and capture your site in a mobile-responsive viewport.
Note: Crawling mobile view adds approximately 1.75x to total crawl time.
Dismiss Pop-ups: Input specific button labels or keywords (e.g., Accept, Decline, Subscribe) separated by Enter. The crawler will click these elements to close cookie banners and modal overlays prior to capturing screenshots.
Whitelist / Allowlist: Enable this if your site has strict security or firewall rules blocking automated traffic. Provide the crawler’s IP address and User-Agent string to your IT or SEO team to ensure seamless access.
Enable Proxy: Routes crawling traffic through proxy servers to prevent requests from being flagged or blocked as automated bot activity.
You should enable a proxy if the target website uses strict rate limiting, IP blocking, or geo-blocking that prevents our standard crawler from accessing its pages. Using a proxy helps bypass these restrictions to ensure all pages are successfully crawled, though it may slightly increase overall crawl time.
Crawl Speed: Select between Cautious and Balanced presets to balance speed against server load.
Manual Rate Limit: Toggle this option to define custom request limits per second or minute.
Click Advanced Settings to expose detailed execution timing and viewport controls:
5 m).