Anti-bot protection

Many protected sites do not block a request outright. They first answer with a check page: a script that inspects the browser, proves it is real and then reloads onto the content. ScrapeIt renders such pages in a real browser and returns the page behind the check instead of the check page itself. When a check cannot be passed from one proxy exit, the request is retried through another exit and a fresh browser identity.

Practical advice

• Use a browser for protected sites. Leave jsRendering on auto or set it to true, and allow a generous timeoutMs: some checks (Kasada among them) only pass in a browser.
• Match the exit to the site. proxyCountry set to the site's audience is the single biggest lever. Timezone and language follow the exit automatically.
• Do not block the vendor's scripts. blockResources, blockAds and blockUrls never block the known vendor hosts, but a blockUrls pattern that matches a site's own check script stops the check from running.
• Read the outcome, not the code. A check page and a refused page can both be large and answer 200. request_metadata.executed_runtime, the attempt list and the page title tell you what came back.

Limits

• A site can let a request through and still treat it differently (fewer results, other prices).
• No provider can pass every site, and a site's thresholds change over time.
• A slow exit can time out on a heavy page even after the check passed; raise timeoutMs or pick a faster proxyType.

Related pages

• Web Scraping API - the general single-URL scraper.
• Scraping at scale without bans - how ScrapeIt keeps large scraping projects unblocked.