How the site is crawled

Embed
Preview of the card
<a href="https://seolyticize.com/docs/ref-get-crawl-settings"><img src="https://seolyticize.com/docs/_api/public/orgs/71cb2b7a-a719-4003-b3ea-6321dc1db8ab/spaces/3bf2dbf7-3faa-473c-ba3b-30588f5f8de4/pages/6874282c-8f49-4d3c-9c74-e7d8ce3dbba0/social-card.svg?v=3e7ae33eec8b" alt="How the site is crawled" width="600" height="315"></a>
POSThttps://seolyticize.com/api/v1/tools/get_crawl_settings

The client's crawl settings: mode (spider, list or sitemap), include and exclude URL patterns, page and depth limits, the user agent sent and the robots.txt group obeyed, speed (requests per second and concurrency), query parameters stripped from discovered URLs, custom extraction rules, JavaScript rendering (off, shells — only pages that look unrendered — or all, up to renderMaxPages a crawl), whether staging credentials are set (names only, never values), whether requests are signed with Web Bot Auth, and the crawl schedule (cadence, day, UTC hour and nextRunAt; weekly on Monday 02:00 UTC unless set_crawl_schedule changed it). Answers 'what does the crawl do differently for this site'. Free.

Parameters

Idempotency-Keystringheader
Any unique string, up to 200 characters. A retry with the same key and body is answered from the first answer (Idempotent-Replayed: true) for a day, so it never buys or writes twice; the same key with another body is refused.

Request body

application/json
clientIdstringrequired
{
  "clientId": "string"
}

Code samples

cURL
curl -X POST "https://seolyticize.com/api/v1/tools/get_crawl_settings" \
  -H "Authorization: Bearer $SEOLYTICIZE_API_KEY" \
  -H "Idempotency-Key: <value>" \
  -H "Content-Type: application/json" \
  -d '{"clientId":"string"}'
TypeScript
Python
Go
PHP
Ruby
Java

Responses

200 The tool's result under data, and its meta.
Example
{
  "data": {},
  "meta": {
    "requestId": "string",
    "costMicros": 0,
    "cached": true,
    "cachedFrom": "string",
    "dataAsOf": "string",
    "sources": [
      "string"
    ],
    "partial": true,
    "next": [
      {
        "tool": "string",
        "args": {},
        "why": "string"
      }
    ],
    "nextCursor": "string",
    "dryRun": true
  }
}
Schema
datarequired
metaobjectrequired
Beside every answer: what it cost, whether it came from cache, how fresh it is, whether it is complete, and what to call next.
Show child parameters
requestIdstringrequired
costMicrosintegerrequired
Charged by this call, micro-USD (1,000,000 = $1).
cachedbooleanrequired
True when nothing was bought.
cachedFromstring
When the oldest cached answer used was bought.
dataAsOfstring
sourcesstring[]
partialbooleanrequired
True when the data is incomplete: a running crawl, unfinished days, unread pages.
nextobject[]required
Show child parameters
toolstring
argsobject
whystring
nextCursorstring
Pass as cursor for the next page.
dryRunboolean
Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
X-Request-Idstring
The id to quote about this request.
400 The body was not JSON or did not match the tool's input schema.
Example
{
  "error": "string",
  "code": "string",
  "next": "string",
  "requestId": "string"
}
Schema
errorstringrequired
A sentence written for the caller. Internal failures are reduced to 'The request could not be completed.' and never carry detail.
codestring
A stable snake_case reason to branch on; GET /api/v1/errors lists them all.
nextstring
The call or step that fixes it, when there is one.
requestIdstring
Quote it when asking about a failure.
Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
401 No key, or a key that is unknown or revoked.

Same as 400 · POST /api/v1/tools/get_crawl_settings

Headers
WWW-Authenticatestring
403 The key's role does not permit the tool, or the key may not reach the named client.

Same as 400 · POST /api/v1/tools/get_crawl_settings

Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
404 No tool by that name.

Same as 400 · POST /api/v1/tools/get_crawl_settings

Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
409 A request with this Idempotency-Key is still running.

Same as 400 · POST /api/v1/tools/get_crawl_settings

Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
422 A well-formed request the product declined, with the reason — a duplicate name, a plan limit, nothing configured to run.

Same as 400 · POST /api/v1/tools/get_crawl_settings

Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
429 This key has used its window.

Same as 400 · POST /api/v1/tools/get_crawl_settings

Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
Retry-Afterinteger
Seconds to wait before retrying.
500 The request could not be completed.

Same as 400 · POST /api/v1/tools/get_crawl_settings

Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
Updated