Page interactions

Some data only appears after you interact with the page: dismiss a cookie banner, click “load more”, run a search, scroll to trigger lazy loading. Set jsRendering: true and describe the steps with jsScenario.

Steps run sequentially, after navigation and before extraction, in a managed remote browser with a hardened fingerprint. The vocabulary is small and explicit on purpose; the one step that runs your own code, evaluate, runs it inside the target page - see below.

Step vocabulary

action	Fields	Example	Purpose
wait	durationMs (0–30000)	{"action":"wait","durationMs":1500}	Fixed pause.
wait_for	selector, timeoutMs? (1–60000)	{"action":"wait_for","selector":".results","timeoutMs":15000}	Block until a CSS selector appears.
click	selector	{"action":"click","selector":"#load-more"}	Click an element (banners, “load more”, tabs).
fill	selector, value	{"action":"fill","selector":"input[name=q]","value":"laptop"}	Type a value into an input.
press	key, selector?	{"action":"press","key":"Enter","selector":"input[name=q]"}	Press a key or chord (Enter, Escape, ArrowDown, Control+A) on selector or the focused element.
select	selector, and value or label (a string or an array)	{"action":"select","selector":"#sort","label":"Price: low to high"}	Choose option(s) of a <select> by their value attribute or by their visible text.
scroll	target: top | bottom	{"action":"scroll","target":"bottom"}	Scroll to trigger lazy-loaded content.
evaluate	expression (up to 4 096 characters), timeoutMs? (1–30000)	{"action":"evaluate","expression":"document.querySelectorAll('.product').length"}	Run a JavaScript expression in the page and return its value in content.scenarioResults.

Limits. Up to 50 steps per request, at most 10 of them evaluate. The scenario runs for at most 120 seconds (and never past the request's own time budget); a scenario whose declared wait durations and timeoutMs already add up to more is rejected. A step without its own timeout gets up to 30 seconds.

A step that fails (a selector that never appears, a key the browser does not know) does not fail the request: the scenario goes on and the page is captured as it is. A malformed scenario is rejected with 400 VALIDATION_ERROR before any work starts; the message starts with a stable reason code: JS_SCENARIO_INVALID_STEP, JS_SCENARIO_TOO_MANY_STEPS, JS_SCENARIO_TOO_MANY_EVALUATE_STEPS or JS_SCENARIO_DURATION_EXCEEDED.

Example: search then wait for results

cURL

curl -X POST "https://api.scrapeit.io/api/v1/scrape/web" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_SCRAPEIT_API_KEY" \
  -d '{
    "url": "https://shop.example.com",
    "jsRendering": true,
    "jsScenario": [
      { "action": "click", "selector": "#cookie-accept" },
      { "action": "fill", "selector": "input[name=q]", "value": "laptop" },
      { "action": "click", "selector": "button[type=submit]" },
      { "action": "wait_for", "selector": ".results-grid", "timeoutMs": 15000 },
      { "action": "scroll", "target": "bottom" },
      { "action": "wait", "durationMs": 1500 }
    ],
    "outputFormat": ["markdown"]
  }'

Node.js

const response = await fetch('https://api.scrapeit.io/api/v1/scrape/web', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'x-api-key': process.env.SCRAPEIT_API_KEY ?? 'YOUR_SCRAPEIT_API_KEY',
  },
  body: JSON.stringify({
    url: 'https://shop.example.com',
    jsRendering: true,
    jsScenario: [
      { action: 'click', selector: '#cookie-accept' },
      { action: 'fill', selector: 'input[name=q]', value: 'laptop' },
      { action: 'click', selector: 'button[type=submit]' },
      { action: 'wait_for', selector: '.results-grid', timeoutMs: 15000 },
      { action: 'scroll', target: 'bottom' },
      { action: 'wait', durationMs: 1500 },
    ],
    outputFormat: ['markdown'],
  }),
});
const { data } = await response.json();

Python

import os
import requests

response = requests.post(
    "https://api.scrapeit.io/api/v1/scrape/web",
    json={
        "url": "https://shop.example.com",
        "jsRendering": True,
        "jsScenario": [
            {"action": "click", "selector": "#cookie-accept"},
            {"action": "fill", "selector": "input[name=q]", "value": "laptop"},
            {"action": "click", "selector": "button[type=submit]"},
            {"action": "wait_for", "selector": ".results-grid", "timeoutMs": 15000},
            {"action": "scroll", "target": "bottom"},
            {"action": "wait", "durationMs": 1500},
        ],
        "outputFormat": ["markdown"],
    },
    headers={"x-api-key": os.environ.get("SCRAPEIT_API_KEY", "YOUR_SCRAPEIT_API_KEY")},
)

Keys and drop-downs

{
  "url": "https://shop.example.com/search",
  "jsScenario": [
    { "action": "fill", "selector": "input[name=q]", "value": "laptop" },
    { "action": "press", "selector": "input[name=q]", "key": "Enter" },
    { "action": "wait_for", "selector": ".results-grid" },
    { "action": "select", "selector": "select#sort", "value": "price-asc" },
    { "action": "select", "selector": "select#brands", "label": ["Aurora", "Borealis"] },
    { "action": "wait_for", "selector": ".results-grid .product-card" }
  ]
}

evaluate: read values from the page

evaluate runs your JavaScript in the target page, like code typed into the browser's DevTools console - never on ScrapeIt's servers. It sees what the page's own scripts see (the DOM, window, the page's cookies in this request's isolated browser session) and nothing of the platform, the proxy or other requests.

• expression is an expression, an arrow function (it is called) or something returning a promise (it is awaited). For several statements wrap them in a function: () => { const rows = document.querySelectorAll('tr'); return rows.length; }.
• The value is serialized as JSON and can be up to 1 MiB; undefined becomes null.
• timeoutMs (default: what is left of the 30-second step budget) bounds the step. A script still running then is stopped, and the page is captured as it is.

{
  "url": "https://app.example.com/product/42",
  "jsScenario": [
    { "action": "wait_for", "selector": "#app .price" },
    { "action": "evaluate", "expression": "window.__INITIAL_STATE__.product" },
    {
      "action": "evaluate",
      "expression": "() => [...document.querySelectorAll('.review')].length"
    },
    {
      "action": "evaluate",
      "expression": "fetch('/api/stock/42').then((response) => response.json())"
    }
  ]
}

content.scenarioResults has one entry per evaluate step, in order; step is the step's index in jsScenario:

{
  "content": {
    "scenarioResults": [
      {
        "step": 1,
        "action": "evaluate",
        "result": { "id": 42, "title": "Aurora 14\" Laptop", "price": 1299 }
      },
      { "step": 2, "action": "evaluate", "result": 128 },
      { "step": 3, "action": "evaluate", "result": null, "error": "STEP_FAILED" }
    ]
  }
}

A step that gave no value carries error:

error	Meaning
STEP_TIMEOUT	The expression ran past its timeout; the script was stopped.
STEP_FAILED	The expression threw, or the page navigated away while it ran.
RESULT_TOO_LARGE	The JSON value is larger than 1 MiB.
RESULT_NOT_SERIALIZABLE	The value has no JSON form (a circular structure, a BigInt, a function).
STEP_SKIPPED	The scenario's time budget ran out before this step started.

Requests your expression makes from the page (fetch, images) pass the same address checks as the page's own requests: private, local and cloud-metadata addresses are blocked.

wait vs waitFor vs jsScenario

Three overlapping tools - pick the simplest that works:

Need	Use
A fixed pause after load	top-level wait (ms)
Wait for one selector, then scrape	top-level waitFor (a CSS selector)
A sequence of actions (click/fill/scroll + waits)	jsScenario

wait/waitFor are convenient shortcuts for the common single-step case; reach for jsScenario when order matters.

Tips

• Start every scenario that hits a consumer site with a click on the cookie/consent button - it blocks content on most EU sites.
• Prefer wait_for over a long fixed wait: it continues the moment the element is ready instead of always paying the full delay.
• After scroll: bottom, add a short wait so lazy-loaded requests can finish before extraction.
• A scenario always runs in the browser - even with jsRendering omitted or false - and is priced as a rendered request.
• With captureWhenReady the scenario still runs, right after that selector appears (the other render steps are skipped). If the selector never appears on a page that is not an anti-bot challenge, the page is captured and billed as it is, the scenario runs on it, and result_metadata.captureWhenReady.satisfied is false - check the selector.

Related pages

• Data extraction - pull fields out once the page is ready.
• Screenshots - capture the page after your scenario runs.
• Request cost - rendering (required here) costs more than a fetch.