Structured data extraction

Skip HTML parsing on your side: describe what you need and get JSON back.

`extractRules`: CSS selector map

extractRules maps your field name → rule. The simplest rule is a CSS selector string: ScrapeIt takes the first matching element and returns its text content; a selector that matches nothing yields null for that field. For attributes, HTML and repeated structures use a rule object.

cURLNode.jsPython

bash

curl -X POST "https://api.scrapeit.io/api/v1/scrape/web" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_SCRAPEIT_API_KEY" \
  -d '{
    "url": "https://shop.example.com/product/42",
    "extractRules": {
      "title": "h1.product-name",
      "price": ".price-box .price:first-child",
      "sku": "[data-testid=sku]",
      "in_stock": ".availability:contains(In stock)"
    },
    "outputFormat": ["json"]
  }'

const response = await fetch('https://api.scrapeit.io/api/v1/scrape/web', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'x-api-key': process.env.SCRAPEIT_API_KEY ?? 'YOUR_SCRAPEIT_API_KEY',
  },
  body: JSON.stringify({
    url: 'https://shop.example.com/product/42',
    extractRules: {
      title: 'h1.product-name',
      price: '.price-box .price:first-child',
      sku: '[data-testid=sku]',
      in_stock: '.availability:contains(In stock)',
    },
    outputFormat: ['json'],
  }),
});
const { data } = await response.json();
console.log(data.content.json.extractRules);

python

import os
import requests

response = requests.post(
    "https://api.scrapeit.io/api/v1/scrape/web",
    json={
        "url": "https://shop.example.com/product/42",
        "extractRules": {
            "title": "h1.product-name",
            "price": ".price-box .price:first-child",
            "sku": "[data-testid=sku]",
            "in_stock": ".availability:contains(In stock)",
        },
        "outputFormat": ["json"],
    },
    headers={"x-api-key": os.environ.get("SCRAPEIT_API_KEY", "YOUR_SCRAPEIT_API_KEY")},
)
print(response.json()["data"]["content"]["json"]["extractRules"])

Response (content.json fragment - note the fields live under the extractRules key):

json

{
  "content": {
    "json": {
      "extractRules": {
        "title": "Aurora 14\" Laptop",
        "price": "€1,299",
        "sku": "SKU-88421",
        "in_stock": "In stock"
      }
    }
  }
}

Supported selector syntax

Feature	Examples
Tag, id, class, universal	`h1`, `#main`, `.price`, `*`
Attributes with operators	`[data-testid=sku]`, `[href^=https]`, `[class~=badge]`, `[src$=.png]`, `[title*=sale]`, `[lang\|=en]`
Combinators	`div > span` (child), `ul li` (descendant), `h2 + p` (adjacent), `h2 ~ p` (sibling)
Pseudo-classes	`:first-child`, `:last-child`, `:nth-child(3)`, `:not(.ad)`, `:contains(text)`, `:has(img)`
Selector groups	`h1, .headline` (first match of either wins)

:contains(...) and :has(...) are the workhorses for messy real-world markup: “the availability row that actually says In stock”, “the card that has an image”.

A selector the engine cannot parse does not fail the request: that field is null.

Rule objects: attributes, HTML and lists

Instead of a string, a field can take an object. A string rule "h1" is the same as { "selector": "h1" }.

Key	Type	Values	Default	Example	Meaning
`selector`	string	CSS selector	required	`.product_pod h3 a`	What to match. Inside a list item it is matched against the item and its descendants, so `"a"` also reads an item that is itself a link.
`type`	string	`item`, `list`	`item`	`list`	`item` - the first match (`null` when none); `list` - every match, as an array (`[]` when none).
`output`	string \| object	`text`, `html`, `attribute` or a map of rules	`text` (`attribute` when `attribute` is set)	`attribute`	`text` - text content, trimmed, whitespace collapsed; `html` - the element's outer HTML; `attribute` - the value of `attribute`; a map of rules - nested fields extracted inside each match.
`attribute`	string	attribute name: `href`, `src`, `data-sku`, …	-	`href`	Attribute to read (case-insensitive). `href` and `src` come back as absolute URLs, resolved against the page address and its `<base href>`. A match without the attribute gives `null`; a list skips it.

A product grid in one request - a list of objects, each with its own fields, links and a nested list:

json

{
  "url": "https://shop.example.com/laptops",
  "extractRules": {
    "heading": "h1",
    "next_page": { "selector": "a.pagination-next", "attribute": "href" },
    "products": {
      "selector": ".product-card",
      "type": "list",
      "output": {
        "name": ".product-name",
        "price": ".price",
        "url": { "selector": "a", "attribute": "href" },
        "image": { "selector": "img", "attribute": "src" },
        "badges": { "selector": ".badge", "type": "list" }
      }
    },
    "description_html": { "selector": ".category-intro", "output": "html" }
  }
}

json

{
  "content": {
    "json": {
      "extractRules": {
        "heading": "Laptops",
        "next_page": "https://shop.example.com/laptops?page=2",
        "products": [
          {
            "name": "Aurora 14\" Laptop",
            "price": "€1,299",
            "url": "https://shop.example.com/product/42",
            "image": "https://cdn.shop.example.com/img/42.jpg",
            "badges": ["New", "Free shipping"]
          },
          {
            "name": "Borealis 16\"",
            "price": "€1,599",
            "url": "https://shop.example.com/product/43",
            "image": null,
            "badges": []
          }
        ],
        "description_html": "<div class=\"category-intro\"><p>Thin and light.</p></div>"
      }
    }
  }
}

An item rule with a map of rules returns one object (the first match) instead of a list: { "selector": ".seller", "output": { "name": ".seller-name" } }.

Limits.

At most 100 rules in total, nested rules included.
At most 3 levels of rule maps: the top level plus two nested output maps (for example list → item → list).
Selectors up to 1 024 characters.
At most 1 000 list values per page, all lists together (nested lists included). Matches past that are dropped.
At most 10 MB of extracted values per page (UTF-8 bytes of the text, HTML and attribute values of all rules together, nested rules included). Values past it are dropped in document order: a list ends early and an item field is null. A type: "list" over nested elements repeats each subtree in every ancestor's value, so prefer the innermost selector.

A request that breaks a limit or the rule shape is rejected with 400 VALIDATION_ERROR before any work starts; the message starts with a stable reason code: EXTRACT_RULES_INVALID (a malformed rule - the message names the field), EXTRACT_RULES_TOO_MANY or EXTRACT_RULES_TOO_DEEP.

Emails and links

Two ready-made extractors - no selectors needed:

json

{ "url": "https://example.com/contacts", "extractEmails": true, "extractLinks": true }

json

{
  "content": {
    "json": {
      "emails": ["sales@example.com", "support@example.com"],
      "links": ["https://example.com/about", "https://example.com/pricing"]
    }
  }
}

emails - deduplicated, lowercased addresses found in the page text.
links - deduplicated absolute http(s) URLs from <a href> (relative hrefs are resolved against the page URL).

Network capture: the SPA shortcut

For single-page apps the cleanest data is in the site's own JSON API calls, not in rendered HTML. With jsRendering: true, captureNetwork records those responses:

json

{
  "url": "https://app.example.com/search?q=laptops",
  "jsRendering": true,
  "captureNetwork": { "enabled": true, "urlIncludes": ["/api/search"], "maxResponses": 10 }
}

json

{
  "content": {
    "networkData": [
      {
        "url": "https://app.example.com/api/search?q=laptops&page=1",
        "status": 200,
        "contentType": "application/json",
        "body": { "items": [{ "id": 1, "title": "Aurora 14\" Laptop", "price": 1299 }] }
      }
    ]
  }
}

Caps: maxResponses ≤ 100, maxBytes ≤ 5 242 880 per body, and the product maxResponses × maxBytes ≤ 16 777 216 bytes (16 MiB) in total - a cap you leave unset counts at its default (20 responses, 524 288 bytes). A request over the total is rejected with 400 VALIDATION_ERROR. Without urlIncludes the default heuristic keeps JSON responses from the target's own domain.

AI extraction (`aiExtractRules`)

Describe the fields you want and an LLM reads the page for you - no selectors. After the scrape succeeds, the page is converted to markdown and sent to the model together with your schema; the answer comes back as ai_extraction, validated against that schema.

json

{
  "url": "https://shop.example/kettle",
  "aiExtractRules": {
    "title": { "type": "string", "required": true, "description": "Product name" },
    "price": { "type": "number", "description": "Current price, without currency" },
    "inStock": { "type": "boolean" },
    "condition": { "type": "enum", "values": ["new", "used"] },
    "variants": { "type": "list", "item": { "type": "string" }, "maxItems": 10 },
    "seller": { "type": "item", "fields": { "name": { "type": "string" } } }
  }
}

json

{
  "ai_extraction": {
    "status": "succeeded",
    "data": {
      "title": "Blue Kettle",
      "price": 39.9,
      "inStock": true,
      "condition": "new",
      "variants": ["1.7 l", "1.2 l"],
      "seller": { "name": "Acme Home" }
    }
  }
}

Field types. string, number, boolean, enum (one of values), list (of item, with optional minItems/maxItems) and item (a nested object with its own fields). description tells the model what to look for. A field the page does not contain is null; mark it required: true when you need a value - an answer without it is rejected. The model can only return these fields with these types; anything else fails validation.

When AI fails. You still get the scrape result (content, files) and ai_extraction is { "status": "failed", "error": { "code": "...", "message": "..." } }, where code is one of scrape_not_successful (the target answered with an error or block page, so there was nothing to read), source_missing, provider_unavailable, timeout, provider_error or invalid_response (the answer did not match the schema).

Price. AI extraction is an add-on on top of the scrape price: +40 credits (request cost); cost.estimated_credits always shows what a request is priced at. Requests with aiExtractRules are never served from the shared cache.

Limits.

The page text sent to the model is capped (60 000 characters by default, roughly 15 000 tokens); a longer page is cut, so put what you need at the top with includeOnlyTags / excludeTags.
The model call has a time budget (30 s by default); it adds to the request's response time.
Schema limits: 8 levels of nesting, 100 fields per object, 200 enum values. Keep schemas shallow - very deep or wide schemas can be refused by the model provider (provider_error).
Available on paid plans (feature ai_extraction); other plans get PLAN_FEATURE_NOT_AVAILABLE.
Works on the Web Scraping API and on batch scraping, where each item's result_metadata.aiExtraction carries the same object. The Website Crawler does not run it yet and rejects aiExtractRules.

Page content is treated as untrusted data: instructions written into a page do not change what the model is asked to do, and the model has no tools - it can only fill your schema.

Output formats - content.json alongside text/markdown.
Page interactions - click/scroll before extracting.
Web Scraping API parameters - the full request reference.

Structured data extraction ​

extractRules: CSS selector map ​

Supported selector syntax ​

Rule objects: attributes, HTML and lists ​

Emails and links ​

Network capture: the SPA shortcut ​

AI extraction (aiExtractRules) ​

Related pages ​

Structured data extraction

`extractRules`: CSS selector map

Supported selector syntax

Rule objects: attributes, HTML and lists

Emails and links

Network capture: the SPA shortcut

AI extraction (`aiExtractRules`)

Related pages