Skip to main content
POST
Fetch contents batch
Give it the addresses as items, each with its own few fields, and at the top the fields every item shares. The results come in input order, each a record or the problem that skipped it. Nothing here is atomic: an item that fails or is skipped leaves the others standing. The batch is not a resource, so it has no id, no expiry and no Location; each record has its own. Use fetch_contents for one address; use upload_contents for a file you hold. A batch takes at most 20 addresses and no files.

What it costs

The key needs contents:fetch, plus contents:model as on fetch_contents, decided over the top fields and every item before any work. The price is the sum of the records’ costs, each priced as a fetch, with units summed by name. An item that was skipped, or whose record failed, was refused or needs a person, bills nothing.

What comes back

200 with ContentsBatch: counts, how the items ended; results, in input order; cost, the sum. Every record is kept 48 hours after its last call and at most 30 days.

The idempotency key

Idempotency-Key is required here, since a repeated batch would fetch and bill every address twice; without it the call is 400 invalid-request naming the header. The whole answer is stored when at least one record was billed, and the same key with the same body returns it again with this call’s request_id, every record’s expires_at as it stands and replayed true on every cost. The same key while the first call still runs is 409 idempotency-key-in-flight; the same key with another body is 422 idempotency-key-reused. A replay whose record was deleted is 404 not-found naming the deletion.

The bounds

  • The body is at most 4 MiB, enough for 20 items with five 4,000-character highlights each in a script such as Chinese, where a character takes three bytes, JSON escaping included.
  • max_characters bounds each record’s text as on a fetch. max_total_characters bounds the whole answer’s text, links counted by their url and text: records take their text in input order while it fits in what remains, and a record that would pass it comes as sections_only with the notice total_characters_cut; get_contents returns that record’s text free unless retention is none. A value below max_characters is allowed.
  • screenshot counts against neither bound and bills one screenshots unit per record that made one; raw_html counts against neither and is free.
  • wait_seconds is each item’s own deadline from its own start. Items run in parallel up to the key’s concurrency and the rest wait inside the call; the whole call ends at bounds.contents.batch_seconds, and an item not started by then is skipped with the problem overloaded and retry_after_seconds set, never deadline.

Money

max_cost is the whole call’s ceiling, reserved at the start in input order: each item its smallest call, one contents, a baseline_fingerprint item too, and the units it asks for by name, the model steps and the screenshot. An item’s record carries the ceiling reserved for it as cost.ceiling. An item the remainder cannot cover for its smallest call is skipped with the problem of the limit that bound it: cost-exceeds-cap for this request’s max_cost, policy-limit-exceeded for the key’s cap, credit-exhausted, run-budget-exhausted or key-budget-exhausted. What remains of the ceiling is shared by the documents’ pages, which spend it as they come; a document stops when it is spent, partial, cost_capped. Inside an item the extras are cut in the order fetch_contents gives, each with the notice cost_cut. json_schema and summary run per record, one model step per record per field.

Faults, and whose they are

  • The batch’s own shape broken, a missing url, a bad or repeated label, a malformed baseline_fingerprint, a bound passed, a misplaced or unknown field: 422 for the whole call, each violation with its pointer, as /items/3/url.
  • A top field’s rule broken: 422 for the whole call, as on fetch_contents.
  • One item’s fault in its address, its domain or a legal demand: that item alone is skipped, its problem beside it, invalid-field, destination-refused, policy-limit-exceeded or legal-block, and the batch runs.
  • One item’s fault learned from the fetch: that item’s record carries the outcome.
  • The same URL twice: two fetches, two records.

Refused before any work

, every violation of the body reported at once in violations:
  • Fewer than 1 or more than 20 items: invalid-field, bound.
  • An item without url: required.
  • A label repeated: duplicate; one outside its character set or length: syntax, bound.
  • url, label, highlights or baseline_fingerprint at the top, or a top-only field inside an item: misplaced.
  • max_total_characters outside its range: bound.
  • Every rule of fetch_contents on the top fields and on an item’s own fields, except the address’s own rules, its scheme and userinfo, the destination and the domain policy, which skip that item alone as said above.
  • An unknown field, at the top or in an item: unknown_field.
  • Idempotency-Key absent: 400 invalid-request.

Notices this operation makes

Every notice of fetch_contents, inside each record; total_characters_cut on a record cut by the whole answer’s bound.

Problem types

For the whole call:
  • 400 invalid-request, The request is invalid.
  • 401 unauthorized, The key is missing or rejected.
  • 403 insufficient-scope, The key lacks a scope this call needs; policy-limit-exceeded, The key’s policy forbids this; credit-exhausted, The credit is exhausted; run-budget-exhausted, The run’s budget is exhausted; key-budget-exhausted, The key’s budget is exhausted.
  • 404 not-found, Not found, on a replay whose record was deleted.
  • 409 idempotency-key-in-flight, The same request is still running.
  • 413 body-too-large, The body is too large.
  • 415 unsupported-media-type, The media type is not supported.
  • 422 invalid-field, A field is invalid; model-work-refused, Model work is refused by the request; cost-exceeds-cap, The cost cap is below the smallest call; version-withdrawn, The version is no longer served; idempotency-key-reused, The idempotency key was reused with another body.
  • 429 rate-limit-exceeded, The rate limit is exceeded; concurrency-limit-exceeded, Too many calls are in flight.
  • 500 internal-error, The service failed.
  • 503 overloaded, The service is overloaded.
Inside a skipped item, each without status: invalid-field, destination-refused, policy-limit-exceeded, legal-block, cost-exceeds-cap, credit-exhausted, run-budget-exhausted, key-budget-exhausted, overloaded.

When a record is not delivered

outcome.message says what may work next, and the table under Outcome.reason gives it for every reason.

Authorizations

Authorization
string
header
required

An API key from https://app.getclean.ai, or an access token issued by https://auth.getclean.ai/realms/getclean, sent as Authorization: Bearer <key> on every call. The scopes a key carries and what a 401 and a 403 say are on the keys and sign-in page.

Headers

Idempotency-Key
string
required

One key per logical call, so a retry is never billed twice.

Required on this operation, where a duplicate would cost the most; without it the call is 400 invalid-request naming the header. The SDKs fill a fresh key when you give none and send the same key on every retry of that call; the command line and the MCP server do the same. Give your own key when your program retries across processes, so the second run finds the first answer.

The form: 1 to 255 visible ASCII characters, used exactly as sent, so a value written as a quoted Structured Field String keeps its quotes as part of the key. That is a stated departure from the IETF draft draft-ietf-httpapi-idempotency-key-header, revision 08, which is not yet a standard; this parameter documents the form in full.

What a repeated key does:

  • The same key with the same body returns the first answer again, with this call's request_id, the current expires_at and replayed true on every cost. Nothing is billed again. On the upload the body is compared by a digest over the document and the fields, and a retried start with the same key is the same job.
  • The same key while the first call still runs: 409 idempotency-key-in-flight, with Retry-After.
  • The same key with another body: 422 idempotency-key-reused.

Answers are stored for 24 hours, per account, for billed answers and running jobs only. A call that may bill is kept in the store from its start, so a second call with its key meets the 409 while it runs.

Required string length: 1 - 255
Pattern: ^[!-~]{1,255}$
traceparent
string

Your own trace id for the call, the W3C Trace Context traceparent header, optional on every operation. getclean logs it with the call, so a request in your traces can be matched to its Request-Id, and never forwards it to a site or a source, a stated departure from W3C Trace Context section 3.4, which asks a vendor to pass it on. A malformed value is ignored, never refused, as the same recommendation asks.

Body

application/json

What you send to fetch_contents_batch: the addresses as items, each with the few fields that are its own, and at the top the fields every item shares, each as fetch_contents defines it. Only items is required. url, label, highlights and baseline_fingerprint live in an item only, and a top-only field inside an item is refused misplaced. An explicit null means the same as leaving the field out. The object is closed: a field it does not list, at the top or in an item, is refused invalid-field, unknown_field.

items
object[]
required

The fetches, 1 to 20 ContentsBatchItem objects, answered in this order. Required. The same URL may stand twice and makes two records.

Required array length: 1 - 20 elements
max_characters
integer
default:20000

The most text to return per record, in characters. 0 returns no text but the map of sections and the facts. Default 20,000; 0, or 1,000 to 1,000,000; a value between 1 and 999 is refused bound.

Required range: 0 <= x <= 1000000
max_total_characters
integer
default:200000

The most text in the whole answer, every record's text and links together, links counted by their url and text. Records take their text in input order while it fits in what remains; a record that would pass it comes as sections_only with the notice total_characters_cut, and get_contents returns its text free. Default 200,000; 1,000 to 2,000,000, a value below max_characters allowed.

Required range: 1000 <= x <= 2000000
wait_seconds
number
default:30

The most seconds each item may work, counted from its own start, as on fetch_contents; the whole call ends at bounds.contents.batch_seconds, and an item not started by then is skipped. 5 to 90; default 30.

Required range: 5 <= x <= 90
max_cost
object

The most the whole call may cost, in the account's currency. It is reserved per item in input order, each item's smallest call and the units it asks for by name, and what remains is one shared sum the documents' pages spend. Default the key's cap on one call. A value below one item's smallest call is refused cost-exceeds-cap, with minimum_cap.

scope
enum<string>
default:main

Which view to return for every item: main, the main content alone, or whole_page, everything on the page. Default main. A document has one view, so whole_page on a document makes the notice field_not_applicable.

Available options:
main,
whole_page
formats
string[]

The formats to return for every item, 0 to 7 of these: markdown, the content as one markdown string; html, as clean HTML; doclang, as DocLang, an XML that keeps each block with its place on the page; json, as Block objects; links, every link with its kind; screenshot, a picture of the whole page; raw_html, the page's bytes exactly as served. [] returns the facts only. Default ["markdown"]. An open set; a repeated value is refused duplicate, a value outside the set not_allowed. doclang, screenshot and raw_html apply to a web page only; on a document each makes the notice field_not_applicable.

Maximum array length: 7

One of the formats the field lists.

Minimum string length: 1
block_kinds
string[]

Keep only blocks of these kinds, in every format and in every model step, for every item: 1 to 10 of text, heading, code, math, divider, interactive, table, image, embed, media, an open set. Default: every kind. An empty list is refused bound.

Required array length: 1 - 10 elements

One of the block kinds the field lists.

Minimum string length: 1
json_schema
object

Your own fields, filled from each record by a model, each value with the words it came from; one model step per record, needs contents:model. As on fetch_contents: a JSON Schema 2020-12 object whose root is type: object with properties, at most 50 KB, refused syntax when it is not valid JSON Schema and not_allowed when its root is not type: object, it has no properties, or its pattern needs backtracking; model-work-refused under deterministic_only, policy-limit-exceeded under a key whose policy forbids model work. Default none.

summary
boolean
default:false

Have a model write a summary of each record, at most 1,500 characters, in the content's language, as on fetch_contents; one model step per record, needs contents:model. Default false. Under deterministic_only it is refused model-work-refused; under a key whose policy forbids model work, policy-limit-exceeded.

deterministic_only
boolean
default:false

Turn off, for every item, every step whose output could differ between two runs of the same input, as on fetch_contents: no judged highlight, no schema fill, no summary, and the publisher's kind only from what getclean already knows. Text recognition on a scanned page still runs and is listed, since it repeats under one version. Default false. A key whose policy forbids model work makes it true, with the notice default_set_by_policy; sending false under such a key is policy-limit-exceeded.

as_of_date
string<date>

Read every page as an archive held it on that day, the latest capture at or before the end of the day, UTC; an item's own as_of_date overrides it. A date not after today in UTC. Default none, the live pages.

Example:

"2026-10-01"

subject_country
string

The country every page is read from, ISO 3166-1 alpha-2, one of the countries getclean can read from, Limits.exits; an item's own subject_country overrides it. Default the key's default_exit. A country getclean cannot read from is refused not_allowed, with the countries it can read from in allowed_values.

Pattern: ^[A-Z]{2}$
Example:

"DE"

subject_language
string

The language to ask every site for, as its Accept-Language, a BCP 47 tag of 1 to 35 characters; an item's own subject_language overrides it. Default none: getclean sends its usual Accept-Language, which source.visitor.language reports.

Required string length: 1 - 35
Example:

"de-DE"

recognition_languages
string[]

The languages to expect on scanned pages, for every item that serves a document: 1 to 10 BCP 47 tags of 1 to 35 characters each, among Limits.recognition_languages; another is refused not_allowed. Default detected, as on fetch_contents.

Required array length: 1 - 10 elements
Required string length: 1 - 35
max_source_price
object

The most you would allow getclean to pay a site that asks a price for its page, for every item, as on fetch_contents. getclean pays no site, whatever the value: a page that asks a price ends that record payment_required, with the price in its cost.source_price. Default none.

retention
enum<string>

How long every record of the batch is kept, hold or none, as on fetch_contents. Default the key's policy.retention; hold is allowed only under a hold policy.

Available options:
hold,
none
pinned_version
string

The version of getclean every item runs under, at most 64 characters; execution.version equals it on every record. Default the current version, Limits.current_version. A version not served, never or no longer, is version-withdrawn, with the versions served.

Required string length: 1 - 64
Example:

"1.0.0"

run_id
string
default:run_default

The run every record belongs to, an existing run of the key. Default the key's default run. An unknown run is refused not_allowed.

Pattern: ^run_(default|[A-Za-z0-9]{22})$
metadata
object

Your own labels, returned on every record as sent and never passed to a site: at most 20 string pairs, keys of 1 to 64 characters, values of 1 to 512. Default none. An empty map is refused bound.

Response

The tally first, then one result per item in input order, each a record or a problem without status, then the sum of the records' costs.

What fetch_contents_batch answers: the tally first, then one result per item in input order, then the sum of the records' costs.

counts
object
required

How the items ended, by outcome kind, with the items skipped before any fetch; the six add up to the number of items.

results
object[]
required

One result per item, in the order of the request.

cost
object
required

The sum of the records' costs, units summed by name; ceiling is the whole call's.

request_id
string
required

This call's id, the same as the Request-Id header.

Pattern: ^req_[A-Za-z0-9]{22}$
Example:

"req_01Jx0V1CDhYS12ULqNUtDm"