Fetch contents batch
Fetch 1 to 20 pages or documents at their URLs in one call. You get one record per address, kept as a single fetch’s is, with a tally first and the sum of the costs last.
items, each with its own few fields, and at the top the fields every item shares. The results come in input order, each a record or the problem that skipped it. Nothing here is atomic: an item that fails or is skipped leaves the others standing. The batch is not a resource, so it has no id, no expiry and no Location; each record has its own. Use fetch_contents for one address; use upload_contents for a file you hold. A batch takes at most 20 addresses and no files.
What it costs
The key needscontents:fetch, plus contents:model as on fetch_contents, decided over the top fields and every item before any work. The price is the sum of the records’ costs, each priced as a fetch, with units summed by name. An item that was skipped, or whose record failed, was refused or needs a person, bills nothing.
What comes back
200 with ContentsBatch: counts, how the items ended; results, in input order; cost, the sum. Every record is kept 48 hours after its last call and at most 30 days.
The idempotency key
Idempotency-Key is required here, since a repeated batch would fetch and bill every address twice; without it the call is 400 invalid-request naming the header. The whole answer is stored when at least one record was billed, and the same key with the same body returns it again with this call’s request_id, every record’s expires_at as it stands and replayed true on every cost. The same key while the first call still runs is 409 idempotency-key-in-flight; the same key with another body is 422 idempotency-key-reused. A replay whose record was deleted is 404 not-found naming the deletion.
The bounds
- The body is at most 4 MiB, enough for 20 items with five 4,000-character highlights each in a script such as Chinese, where a character takes three bytes, JSON escaping included.
max_charactersbounds each record’s text as on a fetch.max_total_charactersbounds the whole answer’s text, links counted by their url and text: records take their text in input order while it fits in what remains, and a record that would pass it comes assections_onlywith the noticetotal_characters_cut;get_contentsreturns that record’s text free unlessretentionisnone. A value belowmax_charactersis allowed.screenshotcounts against neither bound and bills onescreenshotsunit per record that made one;raw_htmlcounts against neither and is free.wait_secondsis each item’s own deadline from its own start. Items run in parallel up to the key’s concurrency and the rest wait inside the call; the whole call ends atbounds.contents.batch_seconds, and an item not started by then isskippedwith the problemoverloadedandretry_after_secondsset, neverdeadline.
Money
max_cost is the whole call’s ceiling, reserved at the start in input order: each item its smallest call, one contents, a baseline_fingerprint item too, and the units it asks for by name, the model steps and the screenshot. An item’s record carries the ceiling reserved for it as cost.ceiling. An item the remainder cannot cover for its smallest call is skipped with the problem of the limit that bound it: cost-exceeds-cap for this request’s max_cost, policy-limit-exceeded for the key’s cap, credit-exhausted, run-budget-exhausted or key-budget-exhausted. What remains of the ceiling is shared by the documents’ pages, which spend it as they come; a document stops when it is spent, partial, cost_capped. Inside an item the extras are cut in the order fetch_contents gives, each with the notice cost_cut. json_schema and summary run per record, one model step per record per field.
Faults, and whose they are
- The batch’s own shape broken, a missing
url, a bad or repeatedlabel, a malformedbaseline_fingerprint, a bound passed, a misplaced or unknown field:422for the whole call, each violation with its pointer, as/items/3/url. - A top field’s rule broken:
422for the whole call, as onfetch_contents. - One item’s fault in its address, its domain or a legal demand: that item alone is
skipped, its problem beside it,invalid-field,destination-refused,policy-limit-exceededorlegal-block, and the batch runs. - One item’s fault learned from the fetch: that item’s record carries the outcome.
- The same URL twice: two fetches, two records.
Refused before any work
, every violation of the body reported at once inviolations:
- Fewer than 1 or more than 20 items:
invalid-field,bound. - An item without
url:required. - A
labelrepeated:duplicate; one outside its character set or length:syntax,bound. url,label,highlightsorbaseline_fingerprintat the top, or a top-only field inside an item:misplaced.max_total_charactersoutside its range:bound.- Every rule of
fetch_contentson the top fields and on an item’s own fields, except the address’s own rules, its scheme and userinfo, the destination and the domain policy, which skip that item alone as said above. - An unknown field, at the top or in an item:
unknown_field. Idempotency-Keyabsent:400 invalid-request.
Notices this operation makes
Every notice offetch_contents, inside each record; total_characters_cut on a record cut by the whole answer’s bound.
Problem types
For the whole call:- 400
invalid-request, The request is invalid. - 401
unauthorized, The key is missing or rejected. - 403
insufficient-scope, The key lacks a scope this call needs;policy-limit-exceeded, The key’s policy forbids this;credit-exhausted, The credit is exhausted;run-budget-exhausted, The run’s budget is exhausted;key-budget-exhausted, The key’s budget is exhausted. - 404
not-found, Not found, on a replay whose record was deleted. - 409
idempotency-key-in-flight, The same request is still running. - 413
body-too-large, The body is too large. - 415
unsupported-media-type, The media type is not supported. - 422
invalid-field, A field is invalid;model-work-refused, Model work is refused by the request;cost-exceeds-cap, The cost cap is below the smallest call;version-withdrawn, The version is no longer served;idempotency-key-reused, The idempotency key was reused with another body. - 429
rate-limit-exceeded, The rate limit is exceeded;concurrency-limit-exceeded, Too many calls are in flight. - 500
internal-error, The service failed. - 503
overloaded, The service is overloaded.
status: invalid-field, destination-refused, policy-limit-exceeded, legal-block, cost-exceeds-cap, credit-exhausted, run-budget-exhausted, key-budget-exhausted, overloaded.
When a record is not delivered
outcome.message says what may work next, and the table under Outcome.reason gives it for every reason.Authorizations
An API key from https://app.getclean.ai, or an access token issued by https://auth.getclean.ai/realms/getclean, sent as Authorization: Bearer <key> on every call. The scopes a key carries and what a 401 and a 403 say are on the keys and sign-in page.
Headers
One key per logical call, so a retry is never billed twice.
Required on this operation, where a duplicate would cost the most; without it the call is 400 invalid-request naming the header. The SDKs fill a fresh key when you give none and send the same key on every retry of that call; the command line and the MCP server do the same. Give your own key when your program retries across processes, so the second run finds the first answer.
The form: 1 to 255 visible ASCII characters, used exactly as sent, so a value written as a quoted Structured Field String keeps its quotes as part of the key. That is a stated departure from the IETF draft draft-ietf-httpapi-idempotency-key-header, revision 08, which is not yet a standard; this parameter documents the form in full.
What a repeated key does:
- The same key with the same body returns the first answer again, with this call's
request_id, the currentexpires_atandreplayedtrue on everycost. Nothing is billed again. On the upload the body is compared by a digest over the document and the fields, and a retried start with the same key is the same job. - The same key while the first call still runs:
409 idempotency-key-in-flight, withRetry-After. - The same key with another body:
422 idempotency-key-reused.
Answers are stored for 24 hours, per account, for billed answers and running jobs only. A call that may bill is kept in the store from its start, so a second call with its key meets the 409 while it runs.
1 - 255^[!-~]{1,255}$Your own trace id for the call, the W3C Trace Context traceparent header, optional on every operation. getclean logs it with the call, so a request in your traces can be matched to its Request-Id, and never forwards it to a site or a source, a stated departure from W3C Trace Context section 3.4, which asks a vendor to pass it on. A malformed value is ignored, never refused, as the same recommendation asks.
Body
What you send to fetch_contents_batch: the addresses as items, each with the few fields that are its own, and at the top the fields every item shares, each as fetch_contents defines it. Only items is required. url, label, highlights and baseline_fingerprint live in an item only, and a top-only field inside an item is refused misplaced. An explicit null means the same as leaving the field out. The object is closed: a field it does not list, at the top or in an item, is refused invalid-field, unknown_field.
The most text in the whole answer, every record's text and links together, links counted by their url and text. Records take their text in input order while it fits in what remains; a record that would pass it comes as sections_only with the notice total_characters_cut, and get_contents returns its text free. Default 200,000; 1,000 to 2,000,000, a value below max_characters allowed.
1000 <= x <= 2000000The most the whole call may cost, in the account's currency. It is reserved per item in input order, each item's smallest call and the units it asks for by name, and what remains is one shared sum the documents' pages spend. Default the key's cap on one call. A value below one item's smallest call is refused cost-exceeds-cap, with minimum_cap.
The formats to return for every item, 0 to 7 of these: markdown, the content as one markdown string; html, as clean HTML; doclang, as DocLang, an XML that keeps each block with its place on the page; json, as Block objects; links, every link with its kind; screenshot, a picture of the whole page; raw_html, the page's bytes exactly as served. [] returns the facts only. Default ["markdown"]. An open set; a repeated value is refused duplicate, a value outside the set not_allowed. doclang, screenshot and raw_html apply to a web page only; on a document each makes the notice field_not_applicable.
7One of the formats the field lists.
1Keep only blocks of these kinds, in every format and in every model step, for every item: 1 to 10 of text, heading, code, math, divider, interactive, table, image, embed, media, an open set. Default: every kind. An empty list is refused bound.
1 - 10 elementsOne of the block kinds the field lists.
1Your own fields, filled from each record by a model, each value with the words it came from; one model step per record, needs contents:model. As on fetch_contents: a JSON Schema 2020-12 object whose root is type: object with properties, at most 50 KB, refused syntax when it is not valid JSON Schema and not_allowed when its root is not type: object, it has no properties, or its pattern needs backtracking; model-work-refused under deterministic_only, policy-limit-exceeded under a key whose policy forbids model work. Default none.
Have a model write a summary of each record, at most 1,500 characters, in the content's language, as on fetch_contents; one model step per record, needs contents:model. Default false. Under deterministic_only it is refused model-work-refused; under a key whose policy forbids model work, policy-limit-exceeded.
Turn off, for every item, every step whose output could differ between two runs of the same input, as on fetch_contents: no judged highlight, no schema fill, no summary, and the publisher's kind only from what getclean already knows. Text recognition on a scanned page still runs and is listed, since it repeats under one version. Default false. A key whose policy forbids model work makes it true, with the notice default_set_by_policy; sending false under such a key is policy-limit-exceeded.
The country every page is read from, ISO 3166-1 alpha-2, one of the countries getclean can read from, Limits.exits; an item's own subject_country overrides it. Default the key's default_exit. A country getclean cannot read from is refused not_allowed, with the countries it can read from in allowed_values.
^[A-Z]{2}$"DE"
The language to ask every site for, as its Accept-Language, a BCP 47 tag of 1 to 35 characters; an item's own subject_language overrides it. Default none: getclean sends its usual Accept-Language, which source.visitor.language reports.
1 - 35"de-DE"
The languages to expect on scanned pages, for every item that serves a document: 1 to 10 BCP 47 tags of 1 to 35 characters each, among Limits.recognition_languages; another is refused not_allowed. Default detected, as on fetch_contents.
1 - 10 elements1 - 35The version of getclean every item runs under, at most 64 characters; execution.version equals it on every record. Default the current version, Limits.current_version. A version not served, never or no longer, is version-withdrawn, with the versions served.
1 - 64"1.0.0"
Response
The tally first, then one result per item in input order, each a record or a problem without status, then the sum of the records' costs.
What fetch_contents_batch answers: the tally first, then one result per item in input order, then the sum of the records' costs.