Concurrency & limits

Concurrent requests

Your subscription plan defines how many scraping requests may run at the same time across the whole workspace (sync requests, batch items and crawler pages all count).

The limit is enforced with a lease before any work starts:

1. Your request arrives and asks for a concurrency slot.
2. If a slot is free, it is leased and the scrape starts.
3. If not, the request is rejected immediately with HTTP 429 and error code CONCURRENCY_LIMIT_EXCEEDED.

A 429-rejected request never queues, never opens a browser session and never charges credits.

{
  "error": {
    "code": "CONCURRENCY_LIMIT_EXCEEDED",
    "message": "Concurrency limit exceeded for the current subscription."
  }
}

Client-side patterns

• Keep an in-process semaphore matching your plan's concurrency, so you spend your slots on requests that will run instead of collecting 429s.
• On 429, retry with exponential backoff + jitter; a slot frees up as soon as any active request finishes.
• For large URL sets prefer Batch Scraping: submit once and let the platform schedule items into your available slots automatically.

Request rate limit

Independently of concurrency, every /api/v1 endpoint (job and file lookups, usage, scrape submissions, …) accepts at most 500 requests per 10-second window from one caller - about 50 requests per second on average, with room for short bursts. The budget is counted:

• per API key, for requests with x-api-key or Authorization: Bearer sk_... - every key has its own budget;
• per user, for dashboard sessions;
• per client IP, for unauthenticated requests - and for requests whose API key turns out to be invalid or revoked.

A key's budget is counted only after the key is verified, so a wrong or made-up key never uses up the budget of a real one. Requests with a key also pass a check on the client IP first: an address that has already spent its budget on failed keys gets 429 for every request until the window resets, valid keys included.

Sign-in and account endpoints (login, registration, password reset, email verification, social sign-in) have their own, stricter limits and do not count here.

A request over the budget is rejected immediately with HTTP 429, error code RATE_LIMIT_EXCEEDED and a Retry-After header with the number of seconds until the window resets (details.retry_after_ms carries the same in milliseconds). A rejected request does no work and is never charged.

HTTP/1.1 429 Too Many Requests
Retry-After: 7

{
  "error": {
    "code": "RATE_LIMIT_EXCEEDED",
    "message": "Too many API requests. Retry after the number of seconds in Retry-After.",
    "details": { "retry_after_ms": 6420 }
  }
}

Polling a job every second for each of your concurrent requests stays well inside the budget. If you hit it, wait for Retry-After before the next request instead of retrying in a loop, poll less often, or use webhooks instead of polling.

Response size

A target's answer is read up to 10 MB (10 485 760 bytes) per request. A rendered page (jsRendering, a screenshot, a site API render) may pull up to 20 MB over the network in total - the document, its scripts and API calls together; rendered pages never download files. A bigger answer is cut at the limit: you get what arrived, and the request is charged like any answer of the target.

Real HTML pages and API responses stay far below these limits. When answers from one domain are cut three times within an hour, your workspace's requests to that domain pause until an hour after the last cut answer: they end at once, uncharged, with non_billable_failure_category: "response_too_large" (a site API answers 502 TARGET_RESPONSE_TOO_LARGE). Check that the URL serves a page and not a file, then retry after the pause.

Other limits

Limit	Where it applies	Value
timeoutMs	Sync Web Scraping API	1 000 – 180 000 ms (default 180 000)
wait	Extra wait after page load	0 – 30 000 ms
Batch size	URLs per batch job	Plan-dependent (max_batch_size), at most 50 000 URLs and 25 MiB of request body
Crawler limit	Pages per crawl job	1 – 10 000
Crawler maxDepth	Link depth from seeds	0 – 10
jsScenario steps	Browser automation steps per request	≤ 50
captureNetwork.maxResponses	Harvested JSON responses	≤ 100
captureNetwork.maxBytes	Per-response body cap	≤ 5 242 880 bytes
maxResponses × maxBytes	Total captured volume per request	≤ 16 777 216 bytes (unset caps count at their defaults, 20 × 524 288)
Monthly credits	All products	Plan-dependent, resets each billing period

The plan grid is in Billing & credits; your workspace's current values (credits, concurrency, batch size) are always visible in the dashboard Pricing and Usage pages.

Related pages

• Billing & credits - plans and credit prices.
• Batch Scraping - the right tool when you hit concurrency ceilings.