Can each AI crawler reach and read the pages that matter

Embed
Preview of the card
<a href="https://seolyticize.com/docs/ref-get-ai-crawler-access"><img src="https://seolyticize.com/docs/_api/public/orgs/71cb2b7a-a719-4003-b3ea-6321dc1db8ab/spaces/3bf2dbf7-3faa-473c-ba3b-30588f5f8de4/pages/a5992fb2-cdc2-439a-bf6c-3cb86cf2b167/social-card.svg?v=3e7ae33eec8b" alt="Can each AI crawler reach and read the pages that matter" width="600" height="315"></a>
POSThttps://seolyticize.com/api/v1/tools/get_ai_crawler_access

Per AI agent — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot — for the client's top pages by Search Console clicks: which robots.txt allows and blocks, whether the server or CDN turns it away when the homepage is fetched under its name, which pages need JavaScript to show their content (from the crawl's rendering, which AI crawlers do not do), and, given an access log, how often it came, how many pages it read and how many requests failed. A verdict (open, partial, blocked) and fixes per agent, its purpose (training, search, user) said. Answers 'can ChatGPT read our site' and 'are we blocking AI search by accident'. Free.

Parameters

Idempotency-Keystringheader
Any unique string, up to 200 characters. A retry with the same key and body is answered from the first answer (Idempotent-Replayed: true) for a day, so it never buys or writes twice; the same key with another body is refused.

Request body

application/json
clientIdstringrequired
pagesinteger
= 50
logstring
An access log in combined format, as analyse_server_log takes it.
= null
{
  "clientId": "string",
  "pages": 50,
  "log": null
}

Code samples

cURL
curl -X POST "https://seolyticize.com/api/v1/tools/get_ai_crawler_access" \
  -H "Authorization: Bearer $SEOLYTICIZE_API_KEY" \
  -H "Idempotency-Key: <value>" \
  -H "Content-Type: application/json" \
  -d '{"clientId":"string","pages":50,"log":null}'
TypeScript
Python
Go
PHP
Ruby
Java

Responses

200 The tool's result under data, and its meta.
Example
{
  "data": {},
  "meta": {
    "requestId": "string",
    "costMicros": 0,
    "cached": true,
    "cachedFrom": "string",
    "dataAsOf": "string",
    "sources": [
      "string"
    ],
    "partial": true,
    "next": [
      {
        "tool": "string",
        "args": {},
        "why": "string"
      }
    ],
    "nextCursor": "string",
    "dryRun": true
  }
}
Schema
datarequired
metaobjectrequired
Beside every answer: what it cost, whether it came from cache, how fresh it is, whether it is complete, and what to call next.
Show child parameters
requestIdstringrequired
costMicrosintegerrequired
Charged by this call, micro-USD (1,000,000 = $1).
cachedbooleanrequired
True when nothing was bought.
cachedFromstring
When the oldest cached answer used was bought.
dataAsOfstring
sourcesstring[]
partialbooleanrequired
True when the data is incomplete: a running crawl, unfinished days, unread pages.
nextobject[]required
Show child parameters
toolstring
argsobject
whystring
nextCursorstring
Pass as cursor for the next page.
dryRunboolean
Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
X-Request-Idstring
The id to quote about this request.
400 The body was not JSON or did not match the tool's input schema.
Example
{
  "error": "string",
  "code": "string",
  "next": "string",
  "requestId": "string"
}
Schema
errorstringrequired
A sentence written for the caller. Internal failures are reduced to 'The request could not be completed.' and never carry detail.
codestring
A stable snake_case reason to branch on; GET /api/v1/errors lists them all.
nextstring
The call or step that fixes it, when there is one.
requestIdstring
Quote it when asking about a failure.
Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
401 No key, or a key that is unknown or revoked.

Same as 400 · POST /api/v1/tools/get_ai_crawler_access

Headers
WWW-Authenticatestring
403 The key's role does not permit the tool, or the key may not reach the named client.

Same as 400 · POST /api/v1/tools/get_ai_crawler_access

Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
404 No tool by that name.

Same as 400 · POST /api/v1/tools/get_ai_crawler_access

Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
409 A request with this Idempotency-Key is still running.

Same as 400 · POST /api/v1/tools/get_ai_crawler_access

Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
422 A well-formed request the product declined, with the reason — a duplicate name, a plan limit, nothing configured to run.

Same as 400 · POST /api/v1/tools/get_ai_crawler_access

Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
429 This key has used its window.

Same as 400 · POST /api/v1/tools/get_ai_crawler_access

Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
Retry-Afterinteger
Seconds to wait before retrying.
500 The request could not be completed.

Same as 400 · POST /api/v1/tools/get_ai_crawler_access

Headers
X-RateLimit-Limitinteger
Requests this key may make per window.
X-RateLimit-Remaininginteger
Requests left in the current window.
X-RateLimit-Resetinteger
Seconds until the current window resets.
Updated