Custom headers & cookies

Sometimes a target needs specific request headers or an authenticated cookie to return the page you want. The Web Scraping API forwards both to the target after security filtering.

Headers

headers is a plain object of header name → value, forwarded to the target request:

{
  "url": "https://api.example.com/data",
  "headers": {
    "Accept-Language": "de-DE,de;q=0.9",
    "Referer": "https://example.com/search",
    "X-Requested-With": "XMLHttpRequest"
  }
}

Some headers are managed for you

Headers that must stay coherent with the proxy and browser persona - User-Agent, TLS/client-hint headers, and hop-by-hop headers - are set by ScrapeIt and cannot be overridden. This keeps your fingerprint consistent so a hand-set User-Agent cannot contradict the IP/TLS and get you blocked. Set the device with the device parameter instead.

Cookies

cookies is a list of { name, value } pairs forwarded to the target - useful for scraping behind a login you already hold a session cookie for:

{
  "url": "https://example.com/account/orders",
  "cookies": [
    { "name": "session_id", "value": "abc123" },
    { "name": "locale", "value": "en_US" }
  ]
}

Both the fetch path and the rendered browser path apply the cookies before requesting the page.

Limits

A request (and every item of a batch, which shares them) accepts at most 50 headers totalling 16 KiB and 50 cookies totalling 16 KiB (UTF-8, name plus value). These are the sizes real browsers and servers accept: most servers reject request headers above 16 KiB, and browsers are required to keep 50 cookies per domain. A request over a limit is rejected with 400 and VALIDATION_ERROR before it runs.

Treat forwarded cookies as secrets

A session cookie is a live credential. Scope it tightly, rotate it, and never paste a real one into the Playground or a shared snippet - the docs and Playground keep everything masked by default. In the API Playground, sensitive fields like cookies are shown as *** in generated code snippets.

Device persona

For the non-rendered (fetch) path, device selects a coherent mobile or desktop fingerprint so a mobile request gets a mobile User-Agent + client hints + TLS, not a random mix:

{ "url": "https://example.com", "device": "mobile" }

Value	Effect
desktop	Desktop browser persona.
mobile	Mobile browser persona (mobile UA + client hints + TLS).
(omitted)	Rotation pool, biased to the proxy tier (a mobile-IP gets a mobile persona).

For rendered requests (jsRendering: true) the device comes from the database-managed browser profile, not this parameter.

Full example

cURL

curl -X POST "https://api.scrapeit.io/api/v1/scrape/web" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_SCRAPEIT_API_KEY" \
  -d '{
    "url": "https://example.com/account",
    "device": "desktop",
    "headers": { "Accept-Language": "en-GB,en;q=0.9" },
    "cookies": [{ "name": "session_id", "value": "abc123" }],
    "outputFormat": ["html"]
  }'

Python

import os
import requests

response = requests.post(
    "https://api.scrapeit.io/api/v1/scrape/web",
    json={
        "url": "https://example.com/account",
        "device": "desktop",
        "headers": {"Accept-Language": "en-GB,en;q=0.9"},
        "cookies": [{"name": "session_id", "value": "abc123"}],
        "outputFormat": ["html"],
    },
    headers={"x-api-key": os.environ.get("SCRAPEIT_API_KEY", "YOUR_SCRAPEIT_API_KEY")},
)

Custom headers and cookies never appear in logs, attempt records or webhook payload previews - they are redacted everywhere the platform stores request data.

Related pages

• Proxy options - country and residential/datacenter routing.
• Page interactions - log in by filling a form instead of forwarding a cookie.