Change how the site is crawled

Changes the client's crawl settings for the next crawl; fields left out are kept. mode: spider (follow links from the home page and sitemap), list (crawl only listUrls, up to 5,000 addresses on the client's domain) or sitemap (crawl only the sitemap's URLs). includePatterns / excludePatterns: wildcards on the path and query ('/blog/', '?sort=', '/.pdf') or full addresses — with includes set, only matching URLs are crawled (the home page is still read to find them). maxPages (up to 20,000) and maxDepth (clicks from the start, up to 50). userAgent: preset seolyticize (default), googlebot-smartphone, googlebot-desktop, bingbot, chrome, or custom with your own string; robots.txt is obeyed as that crawler. requestsPerSecond (0.2–10, default 4) and concurrency (1–5); the crawler still backs off on 429s, 5xx and Retry-After. For staging sites' passwords, headers and cookies use set_crawl_auth. stripParams removes query parameters from every discovered URL (names, or prefixes like utm_*), so sorted and filtered copies of a page are crawled once; names match whatever their case (returnTo strips ?returnTo= and ?returnto=), so they are saved lower-cased. extractions reads values off every page (up to 10): css (a selector, optionally an attribute), xpath (//div[@class='author'], //meta[@property='og:type']/@content, (//a)[1]/@href — child and descendant steps, attribute and text tests, positions), regex (first group), jsonld ('Article.datePublished') or meta (a name or property) — see get_extraction_results. render runs JavaScript: off, shells (pages that look unrendered) or all, capped at renderMaxPages. Answers 'crawl this site this way'. Free.