Async requests & `webhookUrl`

A synchronous request holds the HTTP connection until the page is scraped - up to timeoutMs (180 s at most). An async request answers at once with a job id; the request then runs in the background, and when it ends ScrapeIt POSTs the outcome to your webhookUrl, signed like every other webhook. You can also poll for it instead.

Use it for slow targets (rendering, anti-bot pages), serverless callers with short execution limits, and pipelines that already consume webhooks.

POST https://api.scrapeit.io/api/v1/scrape/web/async
x-api-key: YOUR_SCRAPEIT_API_KEY      (scope: scrape:web)
Content-Type: application/json

1. Create the request webhook signing secret (once)

webhookUrl deliveries are signed with your workspace's request webhook signing secret. Create it once on the dashboard Webhooks page (Request webhook secret, workspace owner or admin); it is shown once - store it like a password. Issuing it again replaces the previous secret immediately, including for deliveries still being retried.

An async request with webhookUrl is refused with 422 WEBHOOK_SECRET_NOT_CONFIGURED until the secret exists. Requests without webhookUrl need no secret.

2. Send the request

The body is the Web Scraping API body plus two fields:

Field	Type	Example	Description
`webhookUrl`	string	`https://hooks.example.com/scrapeit`	Public `http(s)` URL that receives the result. Optional: without it, poll for the result.
`webhookEvents`	string[]	`["scrape.request.finished"]`	`scrape.request.finished`, `scrape.request.failed` or both (default). Requires `webhookUrl`. Send only `finished` to skip failure calls.

cache and sessionId are not accepted here (400 VALIDATION_ERROR): an async request always scrapes the target, and sticky sessions apply to synchronous requests only. Every other option - jsRendering, proxies, extractRules, aiExtractRules, screenshot, jsScenario - works as on the synchronous API.

bash

curl -X POST "https://api.scrapeit.io/api/v1/scrape/web/async" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_SCRAPEIT_API_KEY" \
  -d '{
    "url": "https://example.com",
    "outputFormat": ["markdown"],
    "webhookUrl": "https://your-app.example.com/scrapeit/webhook"
  }'

const response = await fetch('https://api.scrapeit.io/api/v1/scrape/web/async', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'x-api-key': process.env.SCRAPEIT_API_KEY ?? 'YOUR_SCRAPEIT_API_KEY',
  },
  body: JSON.stringify({
    url: 'https://example.com',
    outputFormat: ['markdown'],
    webhookUrl: 'https://your-app.example.com/scrapeit/webhook',
  }),
});
const { data: job } = await response.json();
console.log(job.id, job.status); // the result arrives on webhookUrl

python

import os
import requests

response = requests.post(
    "https://api.scrapeit.io/api/v1/scrape/web/async",
    json={
        "url": "https://example.com",
        "outputFormat": ["markdown"],
        "webhookUrl": "https://your-app.example.com/scrapeit/webhook",
    },
    headers={"x-api-key": os.environ.get("SCRAPEIT_API_KEY", "YOUR_SCRAPEIT_API_KEY")},
)
job = response.json()["data"]
print(job["id"], job["status"])  # the result arrives on webhookUrl

php

<?php

$curl = curl_init('https://api.scrapeit.io/api/v1/scrape/web/async');
curl_setopt_array($curl, [
    CURLOPT_POST => true,
    CURLOPT_HTTPHEADER => [
        'Content-Type: application/json',
        'x-api-key: ' . getenv('SCRAPEIT_API_KEY'),
    ],
    CURLOPT_POSTFIELDS => json_encode([
        'url' => 'https://example.com',
        'outputFormat' => ['markdown'],
        'webhookUrl' => 'https://your-app.example.com/scrapeit/webhook',
    ]),
    CURLOPT_RETURNTRANSFER => true,
]);
$body = curl_exec($curl);
if ($body === false) {
    throw new RuntimeException(curl_error($curl));
}
$status = curl_getinfo($curl, CURLINFO_RESPONSE_CODE);
if ($status >= 300) {
    throw new RuntimeException("HTTP $status: $body");
}

$job = json_decode($body, true)['data'];
// 201: the job is queued; the result is POSTed to webhookUrl,
// or poll GET https://api.scrapeit.io/api/v1/scrape/web/async/{id}.
echo $job['id'], PHP_EOL;

java

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.Objects;

public class ScrapeWebAsyncExample {
    public static void main(String[] args) throws Exception {
        String apiKey = Objects.requireNonNull(System.getenv("SCRAPEIT_API_KEY"), "Set SCRAPEIT_API_KEY");
        String payload = "{\"url\":\"https://example.com\",\"outputFormat\":[\"markdown\"],"
            + "\"webhookUrl\":\"https://your-app.example.com/scrapeit/webhook\"}";

        HttpRequest request = HttpRequest.newBuilder(URI.create("https://api.scrapeit.io/api/v1/scrape/web/async"))
            .header("Content-Type", "application/json")
            .header("x-api-key", apiKey)
            .POST(HttpRequest.BodyPublishers.ofString(payload))
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());

        // 201 {"data":{"id":"<job id>","status":"running",...}} - the result is POSTed to webhookUrl,
        // or poll GET https://api.scrapeit.io/api/v1/scrape/web/async/{id}.
        System.out.println(response.statusCode());
        System.out.println(response.body());
    }
}

csharp

using System;
using System.Net.Http;
using System.Text;
using System.Text.Json;

var apiKey = Environment.GetEnvironmentVariable("SCRAPEIT_API_KEY")
    ?? throw new InvalidOperationException("Set SCRAPEIT_API_KEY");
var payload = JsonSerializer.Serialize(new
{
    url = "https://example.com",
    outputFormat = new[] { "markdown" },
    webhookUrl = "https://your-app.example.com/scrapeit/webhook",
});

using var client = new HttpClient();
using var request = new HttpRequestMessage(HttpMethod.Post, "https://api.scrapeit.io/api/v1/scrape/web/async");
request.Headers.Add("x-api-key", apiKey);
request.Content = new StringContent(payload, Encoding.UTF8, "application/json");

using var response = await client.SendAsync(request);
var body = await response.Content.ReadAsStringAsync();
if (!response.IsSuccessStatusCode)
{
    throw new HttpRequestException($"HTTP {(int)response.StatusCode}: {body}");
}

using var document = JsonDocument.Parse(body);
// 201: the job is queued; the result is POSTed to webhookUrl,
// or poll GET https://api.scrapeit.io/api/v1/scrape/web/async/{id}.
Console.WriteLine(document.RootElement.GetProperty("data").GetProperty("id").GetString());

ruby

require 'json'
require 'net/http'

uri = URI('https://api.scrapeit.io/api/v1/scrape/web/async')
request = Net::HTTP::Post.new(uri)
request['Content-Type'] = 'application/json'
request['x-api-key'] = ENV.fetch('SCRAPEIT_API_KEY')
request.body = JSON.generate({
  url: 'https://example.com',
  outputFormat: ['markdown'],
  webhookUrl: 'https://your-app.example.com/scrapeit/webhook',
})

response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) { |http| http.request(request) }
raise "HTTP #{response.code}: #{response.body}" unless response.is_a?(Net::HTTPSuccess)

job = JSON.parse(response.body)['data']
# 201: the job is queued; the result is POSTed to webhookUrl,
# or poll GET https://api.scrapeit.io/api/v1/scrape/web/async/{id}.
puts job['id']

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"os"
)

func main() {
	payload, err := json.Marshal(map[string]any{
		"url":          "https://example.com",
		"outputFormat": []string{"markdown"},
		"webhookUrl":   "https://your-app.example.com/scrapeit/webhook",
	})
	if err != nil {
		panic(err)
	}

	request, err := http.NewRequest(http.MethodPost, "https://api.scrapeit.io/api/v1/scrape/web/async", bytes.NewReader(payload))
	if err != nil {
		panic(err)
	}
	request.Header.Set("Content-Type", "application/json")
	request.Header.Set("x-api-key", os.Getenv("SCRAPEIT_API_KEY"))

	response, err := http.DefaultClient.Do(request)
	if err != nil {
		panic(err)
	}
	defer response.Body.Close()

	body, err := io.ReadAll(response.Body)
	if err != nil {
		panic(err)
	}
	if response.StatusCode >= http.StatusMultipleChoices {
		panic(fmt.Sprintf("HTTP %d: %s", response.StatusCode, body))
	}

	var result struct {
		Data struct {
			ID     string `json:"id"`
			Status string `json:"status"`
		} `json:"data"`
	}
	if err := json.Unmarshal(body, &result); err != nil {
		panic(err)
	}
	// 201: the job is queued; the result is POSTed to webhookUrl,
	// or poll GET https://api.scrapeit.io/api/v1/scrape/web/async/{id}.
	fmt.Println(result.Data.ID, result.Data.Status)
}

The answer is 201 with the job:

json

{
  "data": {
    "id": "b3c1…",
    "status": "running",
    "product_code": "web_scraping_api",
    "counters": { "total": 1, "processed": 0, "credits_spent": 0, "…": 0 },
    "shared_config": { "outputFormat": ["markdown"] },
    "correlation_id": "…",
    "created_at": "2026-10-24T10:00:00.000Z",
    "finished_at": null,
    "result": {
      "id": "…",
      "url": "https://example.com",
      "status": "queued",
      "credits_spent": 0,
      "…": "…"
    }
  }
}

The request is checked exactly like a synchronous one before it is accepted - target URL policy, credits, plan - so a 4xx here means nothing was queued. A rejected webhookUrl returns 422 with no job created.

3. Receive the result

When the request ends, ScrapeIt POSTs one delivery to webhookUrl:

scrape.request.finished - the target was reached and the result persisted, including target HTTP 4xx/5xx responses;
scrape.request.failed - the request ended without reaching the target, or the job was stopped.

data.reason says why the request ended: completed (it got its result), failed (it did not - errorCode says why), stopped_by_user, insufficient_credits or organization_blocked.

json

{
  "id": "5f0b7c9e-…",
  "type": "scrape.request.finished",
  "created_at": "2026-10-24T10:00:07.000Z",
  "data": {
    "jobId": "b3c1…",
    "scrapeRequestId": "8e2d…",
    "productCode": "web_scraping_api",
    "targetUrl": "https://example.com",
    "status": "finished",
    "reason": "completed",
    "executionStatus": "billable_success",
    "billingStatus": "charged",
    "targetHttpStatus": 200,
    "creditsCharged": 4,
    "errorCode": null,
    "resultUrl": "/api/v1/scrape/web/async/b3c1…",
    "rawFileId": "0a9f…",
    "rawContentUrl": "/api/v1/files/0a9f…/raw",
    "screenshotFileId": null,
    "screenshotUrl": null,
    "contentFileId": "c41e…",
    "contentUrl": "/api/v1/files/c41e…/raw"
  }
}

The delivery carries the outcome and links, not the page itself, so it stays small however heavy the page was. Links are paths on https://api.scrapeit.io and need your API key: resultUrl returns the job with its result (scrape:web); the file links stream the whole file (files:read):

rawContentUrl - the page's raw HTML;
contentUrl - a JSON file with everything a synchronous request returns in data.content besides the raw page: json (extractRules, extractEmails, extractLinks), text, markdown, networkData, scenarioResults, and html when includeOnlyTags / excludeTags / removeBase64Images rewrote it. null when the request asked for the page only;
screenshotUrl - the screenshot (PNG or JPEG) when screenshot was requested.

The AI result of aiExtractRules is in the job result's result_metadata.aiExtraction.

Headers, signature and retries are those of every webhook: x-scrapeit-signature, x-scrapeit-timestamp, x-scrapeit-event-id and x-scrapeit-event-type. Answer any 2xx within 10 seconds. Verify the signature with the request webhook signing secret - code for Node.js, Python, PHP, Java, C#, Ruby and Go is in Verifying the signature - reject stale timestamps, and deduplicate on x-scrapeit-event-id: delivery is at least once.

A failed delivery is retried after 30 seconds, 2 minutes and 10 minutes, then dead-lettered; a receiver that keeps failing is paused briefly between attempts. Every delivery is listed in GET /api/v1/webhooks/deliveries and in the job's diagnostics in the dashboard, and can be retried manually from there.

Polling instead

Without webhookUrl - or as a fallback - poll the job until status is finished, failed or stopped:

bash

curl "https://api.scrapeit.io/api/v1/scrape/web/async/JOB_ID" \
  -H "x-api-key: YOUR_SCRAPEIT_API_KEY"

data.result holds the outcome: execution_status, billing_status, target_http_status, credits_spent, error_code and result_metadata (with rawFileId, contentFileId and screenshotFileId). Download a file with GET /api/v1/files/{fileId}/raw (files:read) - the whole file, streamed:

bash

curl "https://api.scrapeit.io/api/v1/files/CONTENT_FILE_ID/raw" \
  -H "x-api-key: YOUR_SCRAPEIT_API_KEY" -o content.json

Wait a few seconds between polls.

Billing, limits and stopping

Priced like a synchronous request. The price is the same flat request cost. The webhook itself is free.
Concurrency. An async request runs as a job of one item and holds one of your plan's concurrent slots while it runs; jobs over the limit are refused with 429 CONCURRENCY_LIMIT_EXCEEDED, as on batches.
No cache. cache is rejected; use the synchronous API for cache hits at half price.
Stopping. POST /api/v1/scrape/batch/web/{id}/stop (scope scrape:batch) stops a request that has not run yet: it ends skipped, and webhookUrl gets scrape.request.failed.
The job also appears in GET /api/v1/jobs and in the dashboard like any batch.

`webhookUrl` rules

Only public http and https URLs: localhost, private, link-local and metadata addresses are refused with 422 VALIDATION_ERROR.
The host is resolved again at every delivery, the connection goes to the address that was checked, and redirects are not followed - a 3xx answer counts as a failed delivery.
A query string is kept, so https://your-app.example.com/hook?order=42 lets you correlate deliveries on your side.

Errors

Code	HTTP	When
`WEBHOOK_SECRET_NOT_CONFIGURED`	422	`webhookUrl` was sent but the request webhook signing secret was never created.
`VALIDATION_ERROR`	400	Unknown field (for example `cache` or `sessionId`), an event other than the two above, `webhookEvents` without `webhookUrl`.
`VALIDATION_ERROR`	422	`webhookUrl` is not a public `http(s)` URL.
`CONCURRENCY_LIMIT_EXCEEDED`	429	All concurrent slots of your plan are busy.
`BATCH_JOB_NOT_FOUND`	404	`GET /api/v1/scrape/web/async/{id}` with an id that is not an async request of your workspace.

Everything else is answered as on the synchronous API.

Web Scraping API - every request parameter.
Webhooks - signature verification in seven languages, retries and delivery logs.
Getting job results - polling vs webhooks for every async product.
Batch Scraping - the same, for thousands of URLs.

Async requests & webhookUrl ​

1. Create the request webhook signing secret (once) ​

2. Send the request ​

3. Receive the result ​

Polling instead ​

Billing, limits and stopping ​

webhookUrl rules ​

Errors ​

Related pages ​

Async requests & `webhookUrl`

1. Create the request webhook signing secret (once)

2. Send the request

3. Receive the result

Polling instead

Billing, limits and stopping

`webhookUrl` rules

Errors

Related pages