Web Scraping API

Q: How do I scrape a website that requires JavaScript?

Set jsRendering to true. The request then runs in a managed remote browser with a hardened, coherent fingerprint instead of a plain HTTP fetch. Add wait or waitFor for late content, jsScenario steps to click, fill and scroll before extraction, and captureNetwork to harvest the JSON APIs a single-page app calls instead of parsing the rendered DOM.

Q: How do I avoid getting blocked while scraping?

Route the request through the right exit: proxyType residential and a proxyCountry matching the audience the site expects. Rendering makes the request look like a real browser, and the persona timezone and locale follow the proxy automatically. When a target blocks an exit, ScrapeIt retries through other healthy proxies of the type you chose inside the same request.

Q: Can I get structured JSON instead of raw HTML?

Yes. Pass extractRules - a map of field name to CSS selector - and the result arrives in content.json; add extractEmails or extractLinks for those specific harvests, or request outputFormat json. Nothing about the page has to be parsed on your side.

Scrape a single URL synchronously: send POST /api/v1/scrape/web with a target url, get the page content back in the same HTTP response. To get a job id at once and the result pushed to your server instead, send the same body to POST /api/v1/scrape/web/async with a webhookUrl.

This page is the full parameter reference. Task-focused guides:

Output formats - HTML, text, markdown or extracted JSON.
Structured data extraction - CSS rules, emails, links, SPA JSON APIs.
Page interactions - click, fill, scroll and wait before scraping.
Custom headers & cookies - forward your own headers, cookies and device persona.
Screenshots - capture the page as a PNG artifact.
Caching - reuse a scraped page at half the credit cost.
Async requests & webhookUrl - run a request in the background and receive the signed result.
Request cost - the credit price of a request.

POST https://api.scrapeit.io/api/v1/scrape/web
x-api-key: YOUR_SCRAPEIT_API_KEY
Content-Type: application/json

Minimal request:

bash

curl -X POST "https://api.scrapeit.io/api/v1/scrape/web" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_SCRAPEIT_API_KEY" \
  -d '{"url": "https://example.com", "outputFormat": ["markdown"]}'

const response = await fetch('https://api.scrapeit.io/api/v1/scrape/web', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'x-api-key': process.env.SCRAPEIT_API_KEY ?? 'YOUR_SCRAPEIT_API_KEY',
  },
  body: JSON.stringify({ url: 'https://example.com', outputFormat: ['markdown'] }),
});
const result = await response.json();
console.log(result.data.content.markdown);

python

import os
import requests

response = requests.post(
    "https://api.scrapeit.io/api/v1/scrape/web",
    json={"url": "https://example.com", "outputFormat": ["markdown"]},
    headers={"x-api-key": os.environ.get("SCRAPEIT_API_KEY", "YOUR_SCRAPEIT_API_KEY")},
)
result = response.json()
print(result["data"]["content"]["markdown"])

php

<?php

$curl = curl_init('https://api.scrapeit.io/api/v1/scrape/web');
curl_setopt_array($curl, [
    CURLOPT_POST => true,
    CURLOPT_HTTPHEADER => [
        'Content-Type: application/json',
        'x-api-key: ' . getenv('SCRAPEIT_API_KEY'),
    ],
    CURLOPT_POSTFIELDS => json_encode([
        'url' => 'https://example.com',
        'outputFormat' => ['markdown'],
    ]),
    CURLOPT_RETURNTRANSFER => true,
]);
$body = curl_exec($curl);
if ($body === false) {
    throw new RuntimeException(curl_error($curl));
}
$status = curl_getinfo($curl, CURLINFO_RESPONSE_CODE);
if ($status >= 300) {
    throw new RuntimeException("HTTP $status: $body");
}

$response = json_decode($body, true);
echo $response['data']['content']['markdown'], PHP_EOL;

java

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.Objects;

public class ScrapeWebExample {
    public static void main(String[] args) throws Exception {
        String apiKey = Objects.requireNonNull(System.getenv("SCRAPEIT_API_KEY"), "Set SCRAPEIT_API_KEY");
        String payload = "{\"url\":\"https://example.com\",\"outputFormat\":[\"markdown\"]}";

        HttpRequest request = HttpRequest.newBuilder(URI.create("https://api.scrapeit.io/api/v1/scrape/web"))
            .header("Content-Type", "application/json")
            .header("x-api-key", apiKey)
            .POST(HttpRequest.BodyPublishers.ofString(payload))
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());

        // {"data":{"content":{"markdown":"..."},...},"meta":{...}} - read data.content.markdown
        // with the JSON library of your project (Jackson, Gson, ...).
        System.out.println(response.statusCode());
        System.out.println(response.body());
    }
}

csharp

using System;
using System.Net.Http;
using System.Text;
using System.Text.Json;

var apiKey = Environment.GetEnvironmentVariable("SCRAPEIT_API_KEY")
    ?? throw new InvalidOperationException("Set SCRAPEIT_API_KEY");
var payload = JsonSerializer.Serialize(new { url = "https://example.com", outputFormat = new[] { "markdown" } });

// The synchronous API can hold a request up to 180 s; HttpClient gives up after 100 s by default.
using var client = new HttpClient { Timeout = TimeSpan.FromSeconds(200) };
using var request = new HttpRequestMessage(HttpMethod.Post, "https://api.scrapeit.io/api/v1/scrape/web");
request.Headers.Add("x-api-key", apiKey);
request.Content = new StringContent(payload, Encoding.UTF8, "application/json");

using var response = await client.SendAsync(request);
var body = await response.Content.ReadAsStringAsync();
if (!response.IsSuccessStatusCode)
{
    throw new HttpRequestException($"HTTP {(int)response.StatusCode}: {body}");
}

using var document = JsonDocument.Parse(body);
Console.WriteLine(document.RootElement.GetProperty("data").GetProperty("content").GetProperty("markdown").GetString());

ruby

require 'json'
require 'net/http'

uri = URI('https://api.scrapeit.io/api/v1/scrape/web')
request = Net::HTTP::Post.new(uri)
request['Content-Type'] = 'application/json'
request['x-api-key'] = ENV.fetch('SCRAPEIT_API_KEY')
request.body = JSON.generate({ url: 'https://example.com', outputFormat: ['markdown'] })

# The synchronous API can hold a request up to 180 s; Net::HTTP stops reading after 60 s by default.
response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true, read_timeout: 200) do |http|
  http.request(request)
end
raise "HTTP #{response.code}: #{response.body}" unless response.is_a?(Net::HTTPSuccess)

puts JSON.parse(response.body).dig('data', 'content', 'markdown')

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"os"
)

func main() {
	payload, err := json.Marshal(map[string]any{
		"url":          "https://example.com",
		"outputFormat": []string{"markdown"},
	})
	if err != nil {
		panic(err)
	}

	request, err := http.NewRequest(http.MethodPost, "https://api.scrapeit.io/api/v1/scrape/web", bytes.NewReader(payload))
	if err != nil {
		panic(err)
	}
	request.Header.Set("Content-Type", "application/json")
	request.Header.Set("x-api-key", os.Getenv("SCRAPEIT_API_KEY"))

	response, err := http.DefaultClient.Do(request)
	if err != nil {
		panic(err)
	}
	defer response.Body.Close()

	body, err := io.ReadAll(response.Body)
	if err != nil {
		panic(err)
	}
	if response.StatusCode >= http.StatusMultipleChoices {
		panic(fmt.Sprintf("HTTP %d: %s", response.StatusCode, body))
	}

	var result struct {
		Data struct {
			Content struct {
				Markdown string `json:"markdown"`
			} `json:"content"`
		} `json:"data"`
	}
	if err := json.Unmarshal(body, &result); err != nil {
		panic(err)
	}
	fmt.Println(result.Data.Content.Markdown)
}

PHP needs ext-curl; Java 11+ (java.net.http); C# .NET 6+; Go 1.18+. The C# and Ruby examples raise the client timeout because a synchronous request can take up to 180 s.

Core parameters

Parameter	Type	Default	Example	Description
`url`	string	- (required)	`https://books.toscrape.com/catalogue/page-2.html`	Absolute `http`/`https` URL of a public target. Private/local addresses are rejected (`SSRF_BLOCKED`), including on redirects.
`outputFormat`	string[]	`["html"]`	`["html", "markdown"]`	Formats returned inline in `content`: `html`, `text`, `markdown`, `json`.
`timeoutMs`	number	`180000`	`60000`	End-to-end budget, 1 000 – 180 000 ms. Expiry ⇒ `SCRAPE_TIMEOUT`.
`device`	string	rotation	`desktop`	`desktop` or `mobile` persona for non-rendered fetches, so the fingerprint matches the proxy IP class.
`headers`	object	-	`{"Accept-Language":"de-DE"}`	Extra request headers forwarded to the target (security-filtered).
`cookies`	array	-	`[{"name":"currency","value":"EUR"}]`	Cookies forwarded to the target (security-filtered), e.g. `[{"name": "session", "value": "…"}]`.

Proxy options

Route every request through the proxy class and geography you need - this is how you unblock region-locked or bot-protected targets.

Parameter	Type	Default	Example	Description
`proxyType`	string	`datacenter`	`residential`	`datacenter` (fast, cheap), `residential` (real-ISP IPs for hard targets) or `auto` (the platform picks - below). Without `proxyType` the request uses datacenter proxies only.
`proxyCountry`	string	auto	`DE`	ISO-3166 alpha-2 code (`US`, `DE`, `GB`, …), case-insensitive: `de` is accepted and normalized to `DE`. The request exits from that country; the browser/fetch persona's timezone and locale follow the proxy automatically.
`sessionId`	string	-	`checkout-flow-42`	Sticky session: up to 64 letters, digits, `_` or `-`. Requests with the same id keep the same exit IP and the session's cookies (below). Synchronous requests only.

json

{
  "url": "https://www.amazon.de/dp/B0EXAMPLE",
  "proxyType": "residential",
  "proxyCountry": "DE",
  "jsRendering": true
}

Proxy types

`proxyType`	Exits	Price
`datacenter`	datacenter	datacenter
`residential`	residential	residential
`auto`	the platform chooses, below	the tier that served the request

With auto, a request starts on datacenter and moves to residential when the site blocks it. request_metadata.credits_breakdown.proxy_tier shows which tier served it. Prices: request cost.

Sticky sessions

Give consecutive requests the same sessionId to look like one visitor: they leave through the same proxy exit (the same IP on a sticky gateway) and share a cookie jar, so cookies the site set on one request are sent with the next. A login, a cart or a paginated search that the site ties to one visitor keeps working across requests.

json

{
  "url": "https://shop.example.com/cart",
  "proxyType": "residential",
  "proxyCountry": "DE",
  "sessionId": "checkout-42"
}

A session lives 10 minutes after its last successful request; after that the same id starts a new session.
The session belongs to your organization: another organization using the same id gets its own session, and the session's cookies never reach your other requests.
Changing proxyType or proxyCountry starts a new session - another exit pool cannot keep the old IP.
If the session's exit stops working, the request retries through another exit as usual, and the session moves to the exit that answered. A request that got no content leaves the session where it was.
Only proxies that can hold one exit across requests serve a session; when none is configured for the chosen type and country, the request is refused with 422 PROXY_PROVIDER_NOT_FOUND. When they exist but are all briefly out (cooling down or at their traffic limit), it gets 503 PROXY_UNAVAILABLE - retry later with the same sessionId.
Two live sessions do not share one exit, even on a gateway that pins its exits by port: a block another session earns does not land on yours.
An organization can hold up to 100 active sessions; a new id beyond that is refused with 429 SCRAPE_SESSION_LIMIT_EXCEEDED until one expires.
Not available for batch requests (400): parallel items cannot share one exit.

Content check

Some sites answer a distrusted IP with an honest-looking 200 that is a stripped page — a login teaser or a "micro-landing" instead of the product. Tell ScrapeIt what the real page looks like and such an answer is retried through another proxy instead of being returned:

Parameter	Type	Example	Description
`expectContent`	string	`product_main`	Literal text the real page always contains, e.g. a class name of the product title. Up to 256 characters.
`minContentBytes`	number	`20000`	Smallest body, in bytes, a real page has (1 – 10 485 760).

If every attempt gets the stripped page, the last one is returned.

JavaScript rendering

Parameter	Type	Default	Example	Description
`jsRendering`	`"auto"` \| boolean	`"auto"` (web and batch)	`"auto"`	`"auto"`: plain HTTP fetch first, a managed browser only when needed (below). `true`: always render in a managed remote browser with a hardened, coherent fingerprint. `false`: start on the plain fetch.
`wait`	number	`3000` on an anti-bot protected page, otherwise `0`	`2000`	Extra wait after load, 0 – 30 000 ms - for late XHR content. An explicit value, `0` included, always wins over the automatic choice.
`waitFor`	string	-	`.product_pod`	CSS selector (or load condition) to wait for before extraction. If it never appears, the page is still captured, and the response carries `result_metadata.waitFor.satisfied: false`.
`waitUntil`	string	`domcontentloaded`	`load`	Page event to wait for after navigation: `domcontentloaded`, `load` (images and styles loaded), or `networkidle` (no network requests for 500 ms). `load` and `networkidle` wait at most 15 seconds; a page that never goes quiet is then read as it stands.
`jsScenario`	step[]	-	`[{"action":"click","selector":"#load-more"},{"action":"wait","durationMs":1000},{"action":"scroll","target":"bottom"}]`	Up to 50 sequential automation steps executed after navigation (below).
`captureWhenReady`	string	-	`.product_main`	CSS selector marking the real content: captured once it appears, after a `jsScenario` if any; other steps skipped (below).
`humanInteraction`	boolean	`true`	`false`	Human-like cursor movement, scrolling and pauses before extraction. `false` skips the step to cut render time.

All of these except jsRendering itself apply only to rendered requests; a plain HTTP request ignores them. On "auto", waitFor and jsScenario (like screenshot and captureNetwork) render the request on their own.

Auto rendering

With jsRendering omitted or "auto", a request starts on the fast plain HTTP fetch and moves to a real browser - inside the same request - only when it has to:

it asks for something only a browser can do: screenshot, waitFor, a jsScenario, captureNetwork;
the fetched page needs JavaScript: an empty page, a single-page-app shell (an empty root node, or a <noscript> "enable JavaScript" notice on a near-empty page), or extractRules that match nothing in the fetched HTML;
the target answers with an anti-bot challenge.

The request costs the fetch price when the fetch answered and the render price when the page was rendered. request_metadata.js_rendering_mode (auto / enabled / disabled) and request_metadata.executed_runtime (fetch / browser) tell you which case it was. true and false behave as before; false still moves to a browser when the fetched page is empty or a shell.

A page that only needed JavaScript is rendered through the same proxy exit as the fetch (the same IP, so a sessionId keeps its exit); an anti-bot challenge moves the browser to another exit.

Capture before a challenge

Some server-rendered sites send the real page first and let a Cloudflare managed challenge replace it a moment later, once its script runs. captureWhenReady races that challenge: as soon as the selector appears, your jsScenario (if any) runs and the page is captured and returned - waitFor, wait and humanInteraction are skipped. If the selector does not appear within about 5 seconds on a page that is an anti-bot challenge, the attempt is retried through another proxy and browser identity; when no attempt finds it, the request ends as anti_bot_challenge. On a page that is not a challenge (a selector with a typo, a page without that element) the page is captured as it is, the jsScenario still runs, and result_metadata.captureWhenReady.satisfied is false. Leave it unset for single-page apps whose content only appears after interaction.

Human-like interaction

With humanInteraction on (the default), a page that carries anti-bot protection gets cursor movement, scrolling and reading pauses before extraction, the behaviour its scripts watch for. A page without protection only gets the scrolling that reveals lazy-loaded content. Set it to false for targets whose content is already in the first document and that need no behavioural clearance.

`jsScenario` steps

json

{
  "url": "https://shop.example.com/catalog",
  "jsRendering": true,
  "jsScenario": [
    { "action": "click", "selector": "#cookie-accept" },
    { "action": "fill", "selector": "input[name=q]", "value": "laptop" },
    { "action": "click", "selector": "button[type=submit]" },
    { "action": "wait_for", "selector": ".results-grid", "timeoutMs": 15000 },
    { "action": "scroll", "target": "bottom" },
    { "action": "wait", "durationMs": 1500 }
  ]
}

Action	Fields	Example	Purpose
`wait`	`durationMs` (≤ 30 000)	`{"action":"wait","durationMs":1500}`	Fixed pause.
`wait_for`	`selector`, `timeoutMs?` (≤ 60 000)	`{"action":"wait_for","selector":".results-grid","timeoutMs":15000}`	Wait until a CSS selector appears.
`click`	`selector`	`{"action":"click","selector":"#cookie-accept"}`	Click an element (dismiss banners, "load more").
`fill`	`selector`, `value`	`{"action":"fill","selector":"input[name=q]","value":"laptop"}`	Type into an input.
`press`	`key`, `selector?`	`{"action":"press","key":"Enter","selector":"input[name=q]"}`	Press a key or chord (`Enter`, `Control+A`).
`select`	`selector`, `value` or `label`	`{"action":"select","selector":"#sort","value":"price-asc"}`	Choose `<select>` option(s) by value or text.
`scroll`	`target`: `top` \| `bottom`	`{"action":"scroll","target":"bottom"}`	Trigger lazy-loaded content.
`evaluate`	`expression`, `timeoutMs?` (≤ 30 000)	`{"action":"evaluate","expression":"document.title"}`	Run JavaScript in the page; its JSON value is returned in `content.scenarioResults`.

At most 10 evaluate steps and 120 seconds per scenario. Details, limits and the result format: Page interactions.

Extraction options

Turn pages into structured data on our side instead of parsing HTML on yours.

Parameter	Type	Example	Description
`extractRules`	object	`{"title":"h1","price":{"selector":".price_color","type":"list"}}`	Map of `field name → rule`: a CSS selector string, or `{ selector, type?: "item" \| "list", output?: "text" \| "html" \| "attribute" \| { nested rules }, attribute? }` for attributes (absolute `href`/`src`), HTML and lists - see Data extraction. The result arrives in `content.json`.
`extractEmails`	boolean	`true`	Collect email addresses found on the page.
`extractLinks`	boolean	`true`	Collect hyperlinks found on the page.
`includeOnlyTags`	string[]	`["main", "article"]`	Keep only these HTML tags in the output (e.g. `["main", "article"]`).
`excludeTags`	string[]	`["script", "style", "footer"]`	Strip these tags (e.g. `["script", "style", "nav"]`).
`removeBase64Images`	boolean	`true`	Strip inline base64 images to shrink the payload.
`aiExtractRules`	object	`{"product_name":{"type":"string","description":"the product title"},"price":{"type":"string","description":"the price with currency"}}`	AI-powered extraction schema; the answer is returned as `ai_extraction`. Paid add-on - see Data extraction.

json

{
  "url": "https://news.example.com/article/42",
  "extractRules": {
    "title": "h1",
    "author": ".byline a",
    "published": "time[datetime]",
    "body": "article .content"
  },
  "outputFormat": ["json"]
}

Resource blocking

Parameter	Type	Example	Description
`blockResources`	boolean	`true`	Skip images, fonts and media in rendered mode; on by default (set `false` to load them). Stylesheets and scripts still load, because layout and page JS drive `waitFor`, interactions and content. Ignored when `screenshot: true`, which needs the images.
`blockAds`	boolean	`true`	Block known advertising and tracking hosts in rendered mode.
`blockUrls`	string[]	`["google-analytics.com","*.woff2"]`	Block subresources whose URL matches these patterns (`*` is a wildcard; a pattern without one matches as a substring). The target URL itself is never blocked.

Anti-bot vendor scripts (Cloudflare, hCaptcha, reCAPTCHA, DataDome, PerimeterX, Akamai, Imperva, Queue-it) are never blocked by any of the three options - without them a protected target does not return content at all.

Screenshots

Parameter	Type	Example	Description
`screenshot`	boolean	`true`	Capture a full-page screenshot artifact; pages taller than 16384 px are captured as the visible viewport instead (why). The response's `files` array then contains a `screenshot` file reference - download it via the Files API.

Cache options

The shared cache serves repeated requests instantly at half the credit cost. Only successful HTTP 200 responses are stored.

Field (`cache.*`)	Type	Default	Example	Description
`enabled`	boolean	product default	`true`	Turn cache lookup/write on or off for this request.
`strategy`	string	`prefer_cache`	`prefer_cache`	`prefer_cache` (use a fresh-enough entry), `fresh` (bypass and re-scrape), `stale_while_revalidate` (accepted; currently behaves like `prefer_cache`).
`maxAgeSeconds`	number	-	`3600`	Oldest acceptable cache entry age.
`shareScope`	string	-	`organization`	Who may reuse the entry: `private` (this workspace request-for-request), `organization`, `global`.

json

{
  "url": "https://example.com/pricing",
  "cache": {
    "enabled": true,
    "strategy": "prefer_cache",
    "maxAgeSeconds": 3600,
    "shareScope": "organization"
  }
}

The response reports the decision in request_metadata.cache_status (hit, miss, …).

Network capture (SPA data shortcut)

For single-page apps the data usually lives in the site's own JSON API calls, not in the HTML. captureNetwork harvests those responses during a rendered request:

Field (`captureNetwork.*`)	Type	Example	Description
`enabled`	boolean	`true`	Master switch. Renders an auto request; ignored with `jsRendering: false`.
`urlIncludes`	string[]	`["/api/"]`	Only capture responses whose URL contains one of these substrings. Default heuristic: JSON from the target's own domain.
`maxResponses`	number	`20`	Cap on captured responses (≤ 100).
`maxBytes`	number	`524288`	Per-response body cap (≤ 5 242 880 bytes).

maxResponses × maxBytes must not exceed 16 777 216 bytes (16 MiB); a cap you leave unset counts at its default (20 responses, 524 288 bytes). A request over the total is rejected with 400 VALIDATION_ERROR.

Captured payloads arrive in content.networkData as { url, status, contentType, body } objects.

Response schema

json

{
  "data": {
    "request_metadata": {
      "request_id": "uuid",
      "product_code": "web_scraping_api",
      "target_url": "https://example.com",
      "correlation_id": "uuid",
      "proxy_attempt_count": 1,
      "browser_session_id": "uuid | null",
      "cache_status": "hit | miss | null",
      "js_rendering_mode": "auto | enabled | disabled",
      "executed_runtime": "fetch | browser | null",
      "proxy_traffic_bytes": "number | null",
      "credits_breakdown": {
        "base": 4,
        "rendering": 16,
        "proxy_traffic": 40,
        "proxy_tier": "residential"
      }
    },
    "content": {
      "html": "…",
      "text": "…",
      "markdown": "…",
      "json": { "…": "extractRules result" },
      "networkData": [{ "url": "…", "status": 200, "contentType": "application/json", "body": {} }]
    },
    "files": [
      { "id": "uuid", "type": "screenshot", "mime_type": "image/png", "size_bytes": 12345 }
    ],
    "cost": { "estimated_credits": 15, "charged_credits": 50 },
    "execution_status": "billable_success | non_billable_failure | rejected",
    "billing_status": "charged | not_charged | refunded",
    "billable_result_category": "target_response_received | … | null",
    "non_billable_failure_category": "proxy_error | … | null",
    "target_http_status": 200
  }
}

Timeouts

When timeoutMs runs out you get 504 SCRAPE_TIMEOUT; retry the request if you still need the page.

Frequently asked questions

How do I scrape a website that requires JavaScript?

Set jsRendering: true. The request then runs in a managed remote browser with a hardened, coherent fingerprint instead of a plain HTTP fetch. Add wait or waitFor for late content, jsScenario steps to click, fill and scroll before extraction, and captureNetwork to harvest the JSON APIs a single-page app calls instead of parsing the rendered DOM.

How do I avoid getting blocked while scraping?

Route the request through the right exit: proxyType: "residential" and a proxyCountry matching the audience the site expects. Rendering (jsRendering: true) makes the request look like a real browser, and the persona's timezone and locale follow the proxy automatically. When a target blocks an exit, ScrapeIt retries through other healthy proxies of the type you chose inside the same request. More on Anti-bot protection.

Can I get structured JSON instead of raw HTML?

Yes. Pass extractRules - a map of field name to CSS selector - and the result arrives in content.json; add extractEmails or extractLinks for those specific harvests, or request outputFormat: ["json"]. Nothing about the page has to be parsed on your side.

How do I scrape thousands of URLs?

Use the Batch Scraping API: one POST carries the whole URL list plus a shared config, and each item is tracked separately. To follow links instead of a known list, use the Website Crawler. Both respect your plan's concurrency limit.

Batch Scraping - the same options applied to thousands of URLs.
Website Crawler - follow links automatically.
Apartments.com Scraper API - ready-made rental listing and property fields instead of your own selectors.
Files & artifacts - download screenshots and raw HTML artifacts.
Error codes - what a failed request returns.
Managed web scraping services - when you would rather receive the data than run the requests, ScrapeIt builds, runs and monitors the scrapers for you.

Web Scraping API ​

Core parameters ​

Proxy options ​

Proxy types ​

Sticky sessions ​

Content check ​

JavaScript rendering ​

Auto rendering ​

Capture before a challenge ​

Human-like interaction ​

jsScenario steps ​

Extraction options ​

Resource blocking ​

Screenshots ​

Cache options ​

Network capture (SPA data shortcut) ​

Response schema ​

Timeouts ​

Frequently asked questions ​

How do I scrape a website that requires JavaScript? ​

How do I avoid getting blocked while scraping? ​

Can I get structured JSON instead of raw HTML? ​

How do I scrape thousands of URLs? ​

Related pages ​

Web Scraping API

Core parameters

Proxy options

Proxy types

Sticky sessions

Content check

JavaScript rendering

Auto rendering

Capture before a challenge

Human-like interaction

`jsScenario` steps

Extraction options

Resource blocking

Screenshots

Cache options

Network capture (SPA data shortcut)

Response schema

Timeouts

Frequently asked questions

How do I scrape a website that requires JavaScript?

How do I avoid getting blocked while scraping?

Can I get structured JSON instead of raw HTML?

How do I scrape thousands of URLs?

Related pages