Webhooks

Webhooks push events to your HTTPS endpoint the moment something happens - a sync request finished, a batch started or completed, a crawl finished or failed - so you never have to poll. They are included in every plan, Free included.

Event types

Event	Fired when
scrape.request.finished	A Web Scraping API request or a site API call answered: the target was reached and the result saved, a cache hit included (data.cacheHit: true). An async request also posts it to its webhookUrl.
scrape.request.failed	A Web Scraping API request or a site API call ended without data: it failed before reaching the target, ran out of time (data.errorCode: SCRAPE_TIMEOUT), or a site API call ended TARGET_BLOCKED - its credits came back, so data.billingStatus is refunded and data.creditsRefunded says how many.
batch.job.started	A batch job's items are queued and it started running.
batch.job.finished	A batch job reached a terminal state. data.reason: completed (every item processed), stopped_by_user, insufficient_credits (the balance ran out) or organization_blocked (the workspace was blocked; the items that had not run are not charged); data.status is the job's terminal status.
scraper.job.started	A crawl or a no-code scraper run started running. data.jobType is crawler or no_code_scraper.
scraper.job.finished	A crawl or a no-code run completed (data.reason: completed), you stopped a crawl (stopped_by_user) or the workspace was blocked (organization_blocked; the pages that had not run are not charged).
scraper.job.failed	A crawl or a no-code run ended without delivering what it was started for. A crawl: data.reason no_initial_urls (none of the start URLs could be queued) or no_successful_pages (no page reached its target) - both with status: failed - or insufficient_credits (the balance ran out mid-crawl, status: stopped). A no-code run: failed with the data.errorCode that ended it (for example TARGET_BLOCKED), or interrupted (a restart of the service cut it off); what it was charged and refunded is in data.creditsSpent and data.creditsRefunded.

One request, one event: a site API call that needs several pages behind the scenes still sends one scrape.request.* with the credits of the whole call. A batch sends batch.job.* only, not one scrape.request.* per URL - read the items from the batch results once batch.job.finished arrives. A no-code run sends scraper.job.* only, not an event per page.

There is no per-page crawler event: collect the pages from the job's results once scraper.job.finished (or scraper.job.failed) arrives.

Delivery format

Events arrive as POST requests with a JSON body in a stable envelope:

{
  "id": "5f0b7c9e-…",
  "type": "batch.job.finished",
  "created_at": "2026-07-08T10:15:00.000Z",
  "data": {
    "batchJobId": "uuid",
    "productCode": "web_scraping_api",
    "status": "finished",
    "reason": "completed",
    "totalItems": 1000,
    "processedItems": 1000,
    "billableSuccessItems": 968,
    "nonBillableFailedItems": 32,
    "skippedItems": 0,
    "creditsSpent": 9680
  }
}

Headers on every delivery:

Header	Content
x-scrapeit-signature	v1=<hex HMAC-SHA256 digest> - verify it (below).
x-scrapeit-timestamp	Unix timestamp (seconds) used in the signature.
x-scrapeit-event-id	Unique delivery id - use as an idempotency key.
x-scrapeit-event-type	Mirrors the body's type.

Respond with any 2xx within 10 seconds to acknowledge. Anything else (including timeouts) schedules a retry.

Verifying the signature

Every delivery is signed with your endpoint's signing secret (whsec…, shown once at endpoint creation, rotatable) - or, for the webhookUrl of an async request, with the workspace's request webhook signing secret. Compute HMAC-SHA256 over "<timestamp>.<raw body>" and compare with the header using a constant-time comparison:

Node.js

import { createHmac, timingSafeEqual } from 'node:crypto';

const MAX_TIMESTAMP_AGE_SECONDS = 300;

// rawBody: the exact request bytes (Buffer) or string, before any JSON parsing.
export function verifyScrapeitWebhook(rawBody, signatureHeader, timestampHeader, signingSecret) {
  if (typeof timestampHeader !== 'string' || !/^\d+$/.test(timestampHeader)) {
    return false;
  }
  const ageSeconds = Math.abs(Math.floor(Date.now() / 1000) - Number(timestampHeader));
  if (ageSeconds > MAX_TIMESTAMP_AGE_SECONDS) {
    return false; // replay protection
  }

  const digest = createHmac('sha256', signingSecret)
    .update(`${timestampHeader}.`)
    .update(rawBody)
    .digest('hex');
  const expected = Buffer.from(`v1=${digest}`);
  const received = Buffer.from(signatureHeader ?? '');

  return received.length === expected.length && timingSafeEqual(received, expected);
}

// Express example: keep the RAW body for verification
// app.post('/scrapeit/webhook', express.raw({ type: 'application/json' }), (req, res) => {
//   const isValid = verifyScrapeitWebhook(
//     req.body, // Buffer
//     req.get('x-scrapeit-signature'),
//     req.get('x-scrapeit-timestamp'),
//     process.env.SCRAPEIT_WEBHOOK_SECRET,
//   );
//   if (!isValid) return res.status(401).end();
//   res.status(200).end(); // ack fast, process async
// });

Python

import hashlib
import hmac
import time
from typing import Optional

MAX_TIMESTAMP_AGE_SECONDS = 300


def verify_scrapeit_webhook(
    raw_body: bytes,
    signature_header: Optional[str],
    timestamp_header: Optional[str],
    signing_secret: str,
) -> bool:
    if not timestamp_header or not (timestamp_header.isascii() and timestamp_header.isdigit()):
        return False
    if abs(time.time() - int(timestamp_header)) > MAX_TIMESTAMP_AGE_SECONDS:
        return False  # replay protection

    signed_payload = timestamp_header.encode() + b"." + raw_body
    expected = "v1=" + hmac.new(signing_secret.encode(), signed_payload, hashlib.sha256).hexdigest()
    return hmac.compare_digest((signature_header or "").encode(), expected.encode())


# Flask example: request.get_data() is the raw body
# @app.post("/scrapeit/webhook")
# def scrapeit_webhook():
#     if not verify_scrapeit_webhook(
#         request.get_data(),
#         request.headers.get("x-scrapeit-signature"),
#         request.headers.get("x-scrapeit-timestamp"),
#         os.environ["SCRAPEIT_WEBHOOK_SECRET"],
#     ):
#         abort(401)
#     return "", 200  # ack fast, process async

PHP

<?php

const SCRAPEIT_MAX_TIMESTAMP_AGE_SECONDS = 300;

function verifyScrapeitWebhook(
    string $rawBody,
    string $signatureHeader,
    string $timestampHeader,
    string $signingSecret
): bool {
    if (preg_match('/\A\d{1,18}\z/', $timestampHeader) !== 1) {
        return false;
    }
    if (abs(time() - (int) $timestampHeader) > SCRAPEIT_MAX_TIMESTAMP_AGE_SECONDS) {
        return false; // replay protection
    }

    $expected = 'v1=' . hash_hmac('sha256', $timestampHeader . '.' . $rawBody, $signingSecret);

    return hash_equals($expected, $signatureHeader);
}

// $isValid = verifyScrapeitWebhook(
//     file_get_contents('php://input'), // raw body
//     $_SERVER['HTTP_X_SCRAPEIT_SIGNATURE'] ?? '',
//     $_SERVER['HTTP_X_SCRAPEIT_TIMESTAMP'] ?? '',
//     (string) getenv('SCRAPEIT_WEBHOOK_SECRET')
// );
// if (!$isValid) {
//     http_response_code(401);
//     exit;
// }
// http_response_code(200); // ack fast, process async

Java

import java.nio.charset.StandardCharsets;
import java.security.GeneralSecurityException;
import java.security.MessageDigest;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;

public final class ScrapeitWebhookVerifier {
    private static final long MAX_TIMESTAMP_AGE_SECONDS = 300;

    public static boolean verifyScrapeitWebhook(
            byte[] rawBody, String signatureHeader, String timestampHeader, String signingSecret)
            throws GeneralSecurityException {
        if (signatureHeader == null || timestampHeader == null || !timestampHeader.matches("\\d{1,18}")) {
            return false;
        }
        long nowSeconds = System.currentTimeMillis() / 1000;
        if (Math.abs(nowSeconds - Long.parseLong(timestampHeader)) > MAX_TIMESTAMP_AGE_SECONDS) {
            return false; // replay protection
        }

        Mac mac = Mac.getInstance("HmacSHA256");
        mac.init(new SecretKeySpec(signingSecret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
        mac.update((timestampHeader + ".").getBytes(StandardCharsets.UTF_8));
        byte[] digest = mac.doFinal(rawBody);

        StringBuilder expected = new StringBuilder("v1=");
        for (byte value : digest) {
            expected.append(String.format("%02x", value));
        }

        return MessageDigest.isEqual(
            expected.toString().getBytes(StandardCharsets.UTF_8),
            signatureHeader.getBytes(StandardCharsets.UTF_8));
    }

    // Spring example: take the raw body as byte[]
    // @PostMapping("/scrapeit/webhook")
    // ResponseEntity<Void> webhook(@RequestBody byte[] rawBody,
    //         @RequestHeader("x-scrapeit-signature") String signature,
    //         @RequestHeader("x-scrapeit-timestamp") String timestamp) throws Exception {
    //     boolean isValid = ScrapeitWebhookVerifier.verifyScrapeitWebhook(
    //         rawBody, signature, timestamp, System.getenv("SCRAPEIT_WEBHOOK_SECRET"));
    //     return ResponseEntity.status(isValid ? 200 : 401).build(); // ack fast, process async
    // }
}

C#

using System;
using System.Globalization;
using System.Security.Cryptography;
using System.Text;

public static class ScrapeitWebhook
{
    private const long MaxTimestampAgeSeconds = 300;

    public static bool VerifyScrapeitWebhook(
        byte[] rawBody, string? signatureHeader, string? timestampHeader, string signingSecret)
    {
        if (signatureHeader is null
            || !long.TryParse(timestampHeader, NumberStyles.None, CultureInfo.InvariantCulture, out var timestamp)
            || Math.Abs(DateTimeOffset.UtcNow.ToUnixTimeSeconds() - timestamp) > MaxTimestampAgeSeconds)
        {
            return false; // malformed or replayed
        }

        var prefix = Encoding.UTF8.GetBytes($"{timestampHeader}.");
        var signedPayload = new byte[prefix.Length + rawBody.Length];
        prefix.CopyTo(signedPayload, 0);
        rawBody.CopyTo(signedPayload, prefix.Length);

        var digest = HMACSHA256.HashData(Encoding.UTF8.GetBytes(signingSecret), signedPayload);
        var expected = Encoding.UTF8.GetBytes("v1=" + Convert.ToHexString(digest).ToLowerInvariant());

        return CryptographicOperations.FixedTimeEquals(expected, Encoding.UTF8.GetBytes(signatureHeader));
    }
}

// ASP.NET Core minimal API example: read the raw body bytes
// app.MapPost("/scrapeit/webhook", async (HttpRequest request) =>
// {
//     using var buffer = new MemoryStream();
//     await request.Body.CopyToAsync(buffer);
//     var isValid = ScrapeitWebhook.VerifyScrapeitWebhook(
//         buffer.ToArray(),
//         request.Headers["x-scrapeit-signature"],
//         request.Headers["x-scrapeit-timestamp"],
//         Environment.GetEnvironmentVariable("SCRAPEIT_WEBHOOK_SECRET")!);
//     return isValid ? Results.Ok() : Results.Unauthorized(); // ack fast, process async
// });

Ruby

require 'openssl'

MAX_TIMESTAMP_AGE_SECONDS = 300

def verify_scrapeit_webhook(raw_body, signature_header, timestamp_header, signing_secret)
  return false unless signature_header.is_a?(String) && timestamp_header.to_s.match?(/\A\d+\z/)
  return false if (Time.now.to_i - timestamp_header.to_i).abs > MAX_TIMESTAMP_AGE_SECONDS # replay

  signed_payload = "#{timestamp_header}.".b + raw_body.b
  expected = "v1=#{OpenSSL::HMAC.hexdigest('SHA256', signing_secret, signed_payload)}"

  expected.bytesize == signature_header.bytesize &&
    OpenSSL.fixed_length_secure_compare(expected, signature_header)
end

# Sinatra example: read the raw body before parsing it
# post '/scrapeit/webhook' do
#   raw_body = request.body.read
#   halt 401 unless verify_scrapeit_webhook(
#     raw_body,
#     request.env['HTTP_X_SCRAPEIT_SIGNATURE'],
#     request.env['HTTP_X_SCRAPEIT_TIMESTAMP'],
#     ENV.fetch('SCRAPEIT_WEBHOOK_SECRET')
#   )
#   status 200 # ack fast, process async
# end

Go

package webhook

import (
	"crypto/hmac"
	"crypto/sha256"
	"encoding/hex"
	"strconv"
	"time"
)

const maxTimestampAgeSeconds = 300

// VerifyScrapeitWebhook reports whether rawBody is a fresh delivery signed with signingSecret.
func VerifyScrapeitWebhook(rawBody []byte, signatureHeader, timestampHeader, signingSecret string) bool {
	timestamp, err := strconv.ParseUint(timestampHeader, 10, 63) // digits only, fits int64
	if err != nil {
		return false
	}
	if age := time.Now().Unix() - int64(timestamp); age > maxTimestampAgeSeconds || age < -maxTimestampAgeSeconds {
		return false // replay protection
	}

	mac := hmac.New(sha256.New, []byte(signingSecret))
	mac.Write([]byte(timestampHeader + "."))
	mac.Write(rawBody)
	expected := "v1=" + hex.EncodeToString(mac.Sum(nil))

	return hmac.Equal([]byte(expected), []byte(signatureHeader))
}

// net/http example: read the raw body before parsing it
// http.HandleFunc("/scrapeit/webhook", func(w http.ResponseWriter, r *http.Request) {
// 	rawBody, err := io.ReadAll(r.Body)
// 	if err != nil || !VerifyScrapeitWebhook(rawBody, r.Header.Get("x-scrapeit-signature"),
// 		r.Header.Get("x-scrapeit-timestamp"), os.Getenv("SCRAPEIT_WEBHOOK_SECRET")) {
// 		w.WriteHeader(http.StatusUnauthorized)
// 		return
// 	}
// 	w.WriteHeader(http.StatusOK) // ack fast, process async
// })

Verify against the raw body

Compute the HMAC over the exact raw request bytes, before any JSON parsing/re-serialization - a reformatted body produces a different digest.

Recommended hardening (the examples above already do the first):

• Reject deliveries whose x-scrapeit-timestamp is more than a few minutes away from your clock (replay protection).
• Deduplicate by x-scrapeit-event-id (the same value as the body's id) - every retry and re-send of a delivery carries the same id. See delivery guarantee.

Retries & dead-lettering

Attempt	When
1	Immediately.
2	30 seconds after a failure.
3	2 minutes after that.
4	10 minutes after that.

After the 4th failed HTTP attempt the delivery is dead-lettered: no more automatic retries, but you can retry manually from the dashboard or via POST /api/v1/webhooks/deliveries/{deliveryId}/retry. Delivery statuses: pending → delivered | failed (retry scheduled) | dead_lettered.

Delivery guarantee: at least once

Deliveries are at least once, not exactly once. A delivery left stranded - for example by a restart between your 2xx and ScrapeIt recording it - is picked up by a periodic sweep and sent again, so your receiver can see the same delivery more than once.

Make the receiver idempotent: store the id from the body (also sent as the x-scrapeit-event-id header) and ignore a delivery whose id you have already processed. The id belongs to one delivery to one endpoint and stays the same across automatic retries, sweeps and manual retries; the same event sent to two of your endpoints arrives with two different ids.

Pausing a failing endpoint

After 5 failed attempts in a row ScrapeIt pauses deliveries to that endpoint instead of hitting it with every event: deliveries wait (they are not lost and do not use up their retry attempts), and after the pause a single trial delivery checks the endpoint. Success resumes normal delivery; another failure pauses it again for longer, from 30 seconds up to 10 minutes. Changing the endpoint's URL or re-enabling it ends the pause: new deliveries go out at once, and the ones already waiting are sent at their next try, shown in next_attempt_at of the delivery log (at most 10 minutes away). A delivery waits for a paused endpoint at most 24 hours after it was created; after that it is dead-lettered and can still be retried manually.

A workspace also gets a fair share of delivery capacity, so a slow receiver of one workspace cannot hold up the others: its excess deliveries simply wait their turn.

Automatic disabling

An endpoint that keeps failing is switched off so it stops piling up retries: after 20 failed delivery attempts in a row, once that failure streak has lasted at least one hour, the endpoint's status becomes disabled. Members of the workspace get an in-app notification and the change is recorded in the activity (audit) log.

A successful delivery ends the streak. After fixing your receiver, re-enable the endpoint with PATCH /api/v1/webhooks/{id} and status: "active" (or from the dashboard) - re-enabling resets the streak, so an old run of failures cannot disable it again on the next attempt.

Blocked workspace

While a workspace is blocked, its deliveries are not sent: each is dead-lettered without an HTTP attempt, with the error message The workspace is blocked; the delivery was not sent. Blocking stops the workspace's running jobs, and the deliveries that report a stopped job are still sent, so your integration learns that the job ended instead of waiting for it: batch.job.finished and scraper.job.* with data.status: "stopped", and the scrape.request.failed of an async request whose job was stopped (data.status: "skipped"). Once the workspace is unblocked, a dead-lettered delivery can be retried manually.

Request webhooks (webhookUrl)

An async Web Scraping API request can name its own receiver: POST /api/v1/scrape/web/async with webhookUrl gets one scrape.request.finished or scrape.request.failed delivery when the request ends, with the outcome and links to the result. No endpoint has to be registered for it.

• Signed with the request webhook signing secret - one per workspace, created and rotated on the dashboard Webhooks page (Request webhook secret) or with POST /api/v1/webhooks/request-signing-secret, shown once. Same headers and HMAC as above, so the same verification code works.
• Same delivery guarantees: retries, pausing a failing receiver, at-least-once with a stable x-scrapeit-event-id, and an entry in the delivery log and in the job's diagnostics. A webhookUrl is never disabled automatically - each request names it again.
• Same URL rules: public http(s) only, re-resolved at delivery, no redirects followed.
• Registered endpoints subscribed to batch.job.* also see the async request's job events: an async request runs as a job of one item.

Managing endpoints

Method & path	Purpose
GET /api/v1/webhooks	List endpoints.
POST /api/v1/webhooks	Create an endpoint (url, subscribedEvents[]). Returns the signing secret once.
PATCH /api/v1/webhooks/{id}	Change URL, subscriptions or status (active/disabled).
DELETE /api/v1/webhooks/{id}	Switch the endpoint off: it stays listed as disabled with its delivery log.
POST /api/v1/webhooks/{id}/rotate-secret	Rotate the signing secret (new secret returned once).
POST /api/v1/webhooks/request-signing-secret	Create or rotate the request webhook signing secret for webhookUrl (returned once).

Endpoint URLs must be public HTTP(S) addresses - localhost and private-network URLs are rejected. Disabled endpoints receive no new deliveries.

Delivery logs

Debug integrations without guessing:

Method & path	Purpose
GET /api/v1/webhooks/deliveries	All deliveries in the workspace (paginated, filterable by status).
GET /api/v1/webhooks/{id}/deliveries	Deliveries of one endpoint.
POST /api/v1/webhooks/deliveries/{id}/retry	Manual redelivery (e.g. after fixing your receiver).

Each delivery record shows event_type, status, attempt_count, next_attempt_at, the receiver's response_status, a truncated response_body_preview, a sanitized payload_preview and the correlation_id linking it to the originating job. The dashboard Webhooks page renders the same log visually.

Related pages

• Batch Scraping and Website Crawler - the jobs that emit these events.
• Getting job results - when to subscribe to these events instead of polling.
• Async requests & webhookUrl - one request, its result pushed to the URL it names.
• OpenAPI & AsyncAPI - machine-readable event contract (/api/asyncapi.yml).