Getting job results

Batch scraping, the Website Crawler and async Web Scraping API requests are asynchronous: creating a job returns a job id immediately, and the work runs in the background. There are two ways to know when it's done and collect the output - and one way to stop it early.

Option A - poll for status

Fetch the job status endpoint until the job reaches a terminal state (finished, failed or stopped), then page through the results.

cURL

# Poll status
curl "https://api.scrapeit.io/api/v1/scrape/batch/web/BATCH_JOB_ID" \
  -H "x-api-key: YOUR_SCRAPEIT_API_KEY"

# When status is terminal, page through results
curl "https://api.scrapeit.io/api/v1/scrape/batch/web/BATCH_JOB_ID/results?page=1&per_page=100" \
  -H "x-api-key: YOUR_SCRAPEIT_API_KEY"

Python

import os
import time
import requests

BASE = "https://api.scrapeit.io/api/v1"
KEY = {"x-api-key": os.environ["SCRAPEIT_API_KEY"]}
TERMINAL = {"finished", "failed", "stopped"}


def wait_for_job(job_id: str, poll_seconds: int = 5) -> dict:
    while True:
        job = requests.get(f"{BASE}/scrape/batch/web/{job_id}", headers=KEY).json()["data"]
        if job["status"] in TERMINAL:
            return job
        time.sleep(poll_seconds)  # back off; don't hammer the endpoint


def all_results(job_id: str) -> list:
    results, page = [], 1
    while True:
        body = requests.get(
            f"{BASE}/scrape/batch/web/{job_id}/results",
            params={"page": page, "per_page": 100},
            headers=KEY,
        ).json()
        results.extend(body["data"])
        if page >= body["meta"]["last_page"]:
            return results
        page += 1


job = wait_for_job("BATCH_JOB_ID")
print(job["status"], job["counters"]["credits_spent"])
print(len(all_results("BATCH_JOB_ID")), "items")

Polling etiquette: wait a few seconds between polls (5–15 s is plenty), and stop as soon as the status is terminal. counters on the status response lets you show live progress without fetching results.

Following a no-code scraper run

A no-code scraper job (GET /api/v1/scrapers/{site}/jobs/{jobId}) reports its progress while it sweeps, so you can show where it is instead of only running:

{
  "data": {
    "id": "7c0f9a51-2a42-4e2f-9d3a-9b9b9f0a1d44",
    "product_code": "apartments_com",
    "status": "running",
    "pages_swept": 2,
    "pages_requested": 5,
    "failed_pages": 0,
    "total_items": 24,
    "credits_spent": 60,
    "input": { "city": "Miami", "state": "FL", "max_pages": 5, "output_format": ["json"] },
    "created_at": "2026-11-04T10:00:00.000Z",
    "finished_at": null,
    "result_json_file_id": null,
    "result_csv_file_id": null,
    "result_xml_file_id": null
  }
}

pages_swept of pages_requested is how far the run is (max_pages of the request, clamped to 20); total_items is the records collected so far, credits_spent what it has cost, and failed_pages counts pages that brought nothing - a sweep stops at the first one and keeps what it already collected. input is the request the run was started with, so a page or a script that opens a finished run has its settings. The result file ids are filled when the run finishes. The dashboard's Scrapers without code page shows the same numbers live while the run goes on.

Option B - subscribe to webhooks (recommended)

Instead of polling, register a webhook endpoint and let ScrapeIt call you when the job finishes. You get the event within seconds of completion and make zero wasted status calls.

Product	Finish event
Batch scraping	batch.job.finished
Website Crawler	scraper.job.finished, or scraper.job.failed when the crawl failed
No-code scraper	scraper.job.finished, or scraper.job.failed when the run failed (data.jobType: no_code_scraper)
Async request	scrape.request.finished / scrape.request.failed, posted to the request's own webhookUrl - no endpoint to register

Flow: create the job → receive the finish event on your endpoint → call the results endpoint once to collect output. Always verify the signature before trusting a delivery, and use the x-scrapeit-event-id header as an idempotency key.

Which should I use?

Prefer webhooks for production: they're push-based, near-instant and cheaper. Use polling for quick scripts, one-off jobs, or environments where you can't expose a public receiver.

Downloading result files

A batch item or an async request keeps its output in files: the raw HTML (rawFileId), a JSON file with the rest of the content - extractRules JSON, text, markdown, network data, scenario results (contentFileId) - and the screenshot (screenshotFileId), all referenced from the item's result_metadata. Stream any of them with the API key that owns it (scope files:read):

curl "https://api.scrapeit.io/api/v1/files/FILE_ID/raw" \
  -H "x-api-key: YOUR_SCRAPEIT_API_KEY" -o result

GET /api/v1/files/{id}/content stays the dashboard's inline text preview (the first 1 MiB of a text file, in JSON).

Stopping a job

Need to cancel early? Stop endpoints move the job to stopping → stopped. In-flight items finish; items still queued end as skipped.

# Batch
curl -X POST "https://api.scrapeit.io/api/v1/scrape/batch/web/BATCH_JOB_ID/stop" \
  -H "x-api-key: YOUR_SCRAPEIT_API_KEY"

# Crawler or bulk scraper job (API key scope scrapers:stop)
curl -X DELETE "https://api.scrapeit.io/api/v1/scrapers/jobs/CRAWLER_JOB_ID" \
  -H "x-api-key: YOUR_SCRAPEIT_API_KEY"

Any job type can also be listed with GET /api/v1/jobs (scrapers:read) and stopped with DELETE /api/v1/scrape/jobs/{jobId} (scrapers:stop; a batch also needs scrape:batch).

Result paging in one place

Both products use the same pagination convention: ?page=&per_page= (max 100), with totals in meta.total / meta.last_page, and optional ?sort=-createdAt. Iterate until page === meta.last_page or data comes back empty.

Related pages

• Batch Scraping - endpoints, lifecycle and per-item results.
• Website Crawler - crawl-specific parameters and result files.
• Webhooks - events, signatures, retries and delivery logs.