Fetch contents
Fetch a web page or a document at a URL. You get its main content with the structure intact, in the formats you ask for, the map of its sections, the facts of the source, a fingerprint and the price, as one record that is kept for later reads.
get_contents, get_section, list_blocks and list_links, is answered from the same copy, free. Use upload_contents for a file you hold, or for a long document that must run as a job; use search when you have a question and no URL yet; use get_contents to read a record you already have.
What it costs
The key needscontents:fetch, plus contents:model when you send json_schema, set summary or send highlights without deterministic_only; getclean decides this from the request before any work, so a missing scope is 403 before any 422. The price is 1 contents, plus document_pages and recognized_pages when the URL serves a document, plus model_steps for each model field that ran, plus screenshots when one was made. An unchanged answer bills the same 1 contents. A fetch that failed, was refused or needs a person bills nothing.
What comes back
201 Created, with Location naming the record, for every outcome kind: a page that could not be read is still a record, whose outcome says what happened and what may work next. The record reads in one order: what happened, the notices to read first, when the fetch ran, the content or the map, the facts of the source, the fingerprint, the price.
The idempotency key
Idempotency-Key is optional here. The same key with the same body returns the first answer again, with this call’s request_id, expires_at as it stands and replayed true on cost; nothing is billed again. The same key while the first call still runs is 409 idempotency-key-in-flight; the same key with another body is 422 idempotency-key-reused. A replay whose record was deleted is 404 not-found naming the deletion.
How the text fits max_characters
- It fits: the text formats come whole; the reason is
whole;truncatedis false. - It does not fit: no text format comes;
sections, the map, comes instead; the reason issections_only;truncatedis true.screenshotandraw_htmlstill come. - It does not fit and a highlight was found: the sections that hold highlights come in document order while they fit; a section larger than what remains is cut at a block edge with its highlighted blocks inside; the map comes too, as with
max_characters: 0, and does not count against the limit; the reason ismatching_sections;truncatedis true;get_sectionhas the rest. max_characters: 0: no text; the map comes; the reason iswhole;truncatedis false;screenshot,raw_htmlandlinksstill come, the links within their cap.- The map always carries level 1 and
s0, however many rows. Deeper levels come one whole level at a time from the top, while their rows fitmax_characters; under0, while the rows number at most 200. The noticesections_cutsays when levels were left out. - Two text formats each carry the whole content, so the answer is twice as long;
max_charactersbounds the content once, and each format carries it. - A sent
max_characterswith no text format bounds nothing and makes the noticefield_not_applicable; the default makes none.links,screenshotandraw_htmlare outside it. - To learn a document’s size and price for one
contentsunit, sendmax_characters: 0withmax_costat the smallest call’s price: no page is parsed beyond the count and the outline.
The formats
block_kinds narrows every format: the text formats and links hold only those blocks, and highlights, json_schema and summary read only those blocks; whole then means all of them fit; counts stays the page’s. scope: whole_page makes the blocks, the sections and the counts cover the whole page, and left_out is []. A highlight is marked <mark id="h1">…</mark> in markdown and html and listed in highlight_ids on blocks; nothing is marked in doclang, links, screenshot or raw_html; with no text returned, the highlights list still comes with its ids and places. Every occurrence of a string is a highlight, in document order, at most 50 per call, with the notice highlights_cut past that. highlights, json_schema and summary read the whole content, or the chosen blocks, whatever max_characters says; the bound limits what is returned, not what they read. summary is at most 1,500 characters, in the content’s language. summary and schema_values carry no link and no image; any a model writes is removed with the notice model_link_removed, and a URL value is kept only when it stands in the record’s links or text.
Each highlight string
- Found
exact: its words stand in one block, after Unicode normalization and whitespace collapse, case kept. Free. - Found
near: its words stand there after case folding, with ellipses and the punctuation at the ends of words removed, punctuation inside a word kept, pieces in order, and the dotted and dotless i of Turkic scripts folded to one. Free. - Not found verbatim: a model picks the blocks that bear on it,
judged. One model step per call, however many strings needed it. - Not found, and nothing bears on it, or
deterministic_only: the noticehighlight_not_found, one per string. The step’s price when a model judged it, free underdeterministic_only.
deterministic_only
Nothing runs whose output could differ between two runs of the same input: highlights are found verbatim only; json_schema and summary are refused as model-work-refused, both fields named in violations; the publisher’s kind comes only from what getclean already knows. Text recognition on a scanned page still runs and is listed, since it repeats under one version. A key whose policy forbids model work refuses json_schema and summary with policy-limit-exceeded and finds highlights verbatim only; deterministic_only: false under such a key is policy-limit-exceeded, since a request never loosens the key.
baseline_fingerprint
On a page: the page is read in full and its fingerprint compared. If it matches, the answer is unchanged: no text, no highlights, no fill, no summary, formats echoed [], while the record keeps the content, which get_contents reads. On a document at a URL: the bytes are fetched; when a record of your account issued the fingerprint you sent and its document_digest equals the bytes’ digest, the answer is unchanged without a parse; otherwise the parse runs and bills its pages, and a matching content fingerprint then answers unchanged. If no record of your account issued the fingerprint, you get the notice fingerprint_unknown, and the fetch runs whole at the full price. A fingerprint matches only under the scope, the version, the configuration and the kind of source that made it; one issued under another makes the notice fingerprint_incomparable, naming which, and the fetch runs whole at the full price. A fingerprint of a record no longer kept is fingerprint_unknown too, so an agent that comes back after its record expired pays the full price and compares fingerprints itself.
as_of_date
The record is an archive copy: source.carrier is archive, source.archive names the copy, and source.archives_consulted names the archives of Limits.archives that answered. The copy is the latest capture at or before 23:59:59 UTC of that day whose archived status was 2xx, so a capture later on that day may come, archive.captured_at showing when; an agent that needs the state before an event asks for the day before. source.next_capture_at comes when a later capture exists, so an agent steps forward capture by capture. No such copy is no_history. subject_country, subject_language, screenshot and max_source_price are not applied to an archive copy, a notice each.
screenshot and raw_html
The screenshot is of the live page only: the whole page at its full width, as PNG, with its digest, by a URL getclean serves for as long as the record lives. It ignores scope and block_kinds. It is taken on a second visit to the page, which may be served another page. raw_html is the page’s bytes exactly as the server sent them, decompressed, before any script ran, on an archive copy as the archive served them, with their digest, their size and the Content-Type sent; by a URL under retention hold, inline as data_base64 under none. It ignores scope, block_kinds and max_characters, and it is free. The served bytes and their digest let you show later exactly what the server sent.
A document at the URL
It is parsed inside the fetch. Pastwait_seconds the pages parsed come back partial, time_capped, with the notice upload_to_finish; past max_cost they come back partial, cost_capped. pages.count and the outline come even when no page was parsed.
Money, country and version
max_cost is reserved at the start against the ceiling: the lowest of max_cost, the key’s cap, the run’s budget, the key’s budget and the balance. It is spent in one order, the read of the page or of the document’s pages first, then json_schema, highlights, summary and screenshot. When the ceiling, a budget or the deadline cannot cover everything, the extras are cut from the end, each cut a notice naming the field, and the pages are stopped last, the record partial. A run’s or a key’s budget that stops a parse is budget_capped, with the notice budget_cut. max_source_price: getclean pays no site, whatever the value; a site that asks a price ends failed, payment_required, with the price in cost.source_price and the protocol named in the message. subject_country is the country the page is read from; a page that says it is not offered there is region_restricted, and a site that refuses the visitor with a block page is blocked. pinned_version: execution.version equals it.
Fields that do not apply to some sources
Each is a noticefield_not_applicable naming the field, never a refusal, so a client that always sends a field still works on every source:
doclang,screenshot,raw_htmlorscope: whole_pageon a URL that serves a document;recognition_languageson a web page;subject_country,subject_language,screenshotormax_source_pricewithas_of_date;- a sent
max_characterswith no text format; block_kindswith nothing to narrow.
Contents says which members each kind of source carries.
Refused before any work
, every violation of the body reported at once inviolations:
scope, aformatsvalue, ablock_kindsvalue orretentionoutside its set:invalid-field,not_allowed.- A duplicate in
formatsorblock_kinds:duplicate. An emptyblock_kindslist:bound; sendformats: []for the facts alone. max_charactersbetween 1 and 999:bound.json_schemaorsummaryunder the request’sdeterministic_only:model-work-refused, both fields named.json_schema,summaryordeterministic_only: falseunder a key whose policy forbids model work:403 policy-limit-exceeded.max_costbelow the smallest call’s price:cost-exceeds-cap, withminimum_cap.- A scheme other than http or https, a password in the URL’s userinfo, or a malformed
baseline_fingerprint:syntax. - localhost, an address that is not globally reachable, or a port the Fetch standard lists as bad:
destination-refused. - A host outside the key’s domain policy, or a URL the key’s
policy.refuse_credential_urlsrefuses:403 policy-limit-exceeded. - A
subject_countrygetclean cannot read from:not_allowed, with the countries it can read from inallowed_values. - A
json_schemathat is not valid JSON Schema 2020-12:syntax. One whose root is nottype: object, that has noproperties, or whosepatternneeds backtracking:not_allowed. - An unknown
run_id:not_allowed.retention: holdunder anonepolicy:403 policy-limit-exceeded. - A missing
url:required. Any other bound passed:bound. An unknown field:unknown_field. pinned_versionnot served, never or no longer:version-withdrawn, with the versions served.
deterministic_only against json_schema and summary. No dependent field refuses; block_kinds with nothing to narrow is a notice. Every other pair is independent.
Notices this operation makes
highlight_not_found, highlights_cut, sections_cut, links_cut, hidden_text_cut, structured_data_cut, size_limit, conversion_mismatch, ad_blocker_failed, not_settled, unread_region, no_main_content, paywall_declared, redirected, requests_blocked, field_not_applicable, cost_cut, budget_cut, time_cut, step_failed, model_link_removed, recognition_uncertain, tracked_changes, upload_to_finish, fingerprint_incomparable, fingerprint_unknown, marks_not_read, archive_unreachable, default_set_by_policy, version_retiring. content_not_rendered is a published code this version does not make.
Problem types
- 400
invalid-request, The request is invalid. - 401
unauthorized, The key is missing or rejected. - 403
insufficient-scope, The key lacks a scope this call needs;policy-limit-exceeded, The key’s policy forbids this;credit-exhausted, The credit is exhausted;run-budget-exhausted, The run’s budget is exhausted;key-budget-exhausted, The key’s budget is exhausted. - 404
not-found, Not found, on a replay whose record was deleted. - 409
idempotency-key-in-flight, The same request is still running. - 413
body-too-large, The body is too large. - 415
unsupported-media-type, The media type is not supported. - 422
invalid-field, A field is invalid;destination-refused, The destination is refused;model-work-refused, Model work is refused by the request;cost-exceeds-cap, The cost cap is below the smallest call;version-withdrawn, The version is no longer served;idempotency-key-reused, The idempotency key was reused with another body. - 429
rate-limit-exceeded, The rate limit is exceeded;concurrency-limit-exceeded, Too many calls are in flight. - 451
legal-block, Withheld under a legal demand. - 500
internal-error, The service failed. - 503
overloaded, The service is overloaded.
When a record is not delivered
outcome.message says what may work next, and the table under Outcome.reason gives it for every reason.Authorizations
An API key from https://app.getclean.ai, or an access token issued by https://auth.getclean.ai/realms/getclean, sent as Authorization: Bearer <key> on every call. The scopes a key carries and what a 401 and a 403 say are on the keys and sign-in page.
Headers
One key per logical call, so a retry is never billed twice.
Optional on this operation; required on upload_contents, fetch_contents_batch and search_batch, where a duplicate would cost the most. The SDKs fill a fresh key when you give none and send the same key on every retry of that call; the command line and the MCP server do the same. Give your own key when your program retries across processes, so the second run finds the first answer.
The form: 1 to 255 visible ASCII characters, used exactly as sent, so a value written as a quoted Structured Field String keeps its quotes as part of the key. That is a stated departure from the IETF draft draft-ietf-httpapi-idempotency-key-header, revision 08, which is not yet a standard; this parameter documents the form in full.
What a repeated key does:
- The same key with the same body returns the first answer again, with this call's
request_id,expires_atas it stands andreplayedtrue on everycost. Nothing is billed again. - The same key while the first call still runs:
409 idempotency-key-in-flight, withRetry-After. - The same key with another body:
422 idempotency-key-reused.
Answers are stored for 24 hours, per account, for billed answers and running jobs only; a free answer is not stored. A call that may bill is kept in the store from its start, so a second call with its key meets the 409 while it runs.
1 - 255^[!-~]{1,255}$Your own trace id for the call, the W3C Trace Context traceparent header, optional on every operation. getclean logs it with the call, so a request in your traces can be matched to its Request-Id, and never forwards it to a site or a source, a stated departure from W3C Trace Context section 3.4, which asks a vendor to pass it on. A malformed value is ignored, never refused, as the same recommendation asks.
Body
What you send to fetch_contents. Only url is required; every other field has a default. Every field is at the top level, and an explicit null means the same as leaving the field out. The object is closed: a field it does not list is refused invalid-field, unknown_field.
The address to fetch, a page or a document; getclean tells which. Required. http or https, at most 2,048 characters. A scheme other than http or https, or a password in the URL's userinfo, is refused invalid-field, syntax; localhost, an address that is not globally reachable or a bad port is destination-refused; a host outside the key's domain policy is policy-limit-exceeded.
1 - 2048"https://example.com/q3"
The most text to return, in characters. 0 returns no text but the map of sections and the facts, which is how you see a document's size before paying for its text. Default 20,000; 0, or 1,000 to 1,000,000; a value between 1 and 999 is refused bound.
0 <= x <= 1000000The formats to return, 0 to 7 of these: markdown, the content as one markdown string; html, as clean HTML; doclang, as DocLang, an XML that keeps each block with its place on the page; json, as Block objects; links, every link with its kind; screenshot, a picture of the whole page; raw_html, the page's bytes exactly as served. [] returns the facts only. Default ["markdown"]. An open set; a repeated value is refused duplicate, a value outside the set not_allowed. doclang, screenshot and raw_html apply to a web page only; on a document each makes the notice field_not_applicable.
7One of the formats the field lists.
1Keep only blocks of these kinds, in every format and in every model step: 1 to 10 of text, heading, code, math, divider, interactive, table, image, embed, media, an open set. Default: every kind. An empty list is refused bound. When nothing would use it, no text format, no links, no highlights, no json_schema and no summary, the notice field_not_applicable says so.
1 - 10 elementsOne of the block kinds the field lists.
1Words or questions to find in the content and mark where they stand, 1 to 5 strings of 1 to 4,000 characters each. Each string is one query, and every place it finds carries its position as query_index. Default none. Sending this without deterministic_only needs contents:model, since a string may need a model. A string found verbatim costs nothing; the strings a model had to judge cost one model step for the whole call.
1 - 5 elements1 - 4000Your own fields, filled from the content by a model, each value with the words it came from; one model step, needs contents:model. A JSON Schema 2020-12 object whose root is type: object with properties, at most 50 KB. Default none. A schema that is not valid JSON Schema is refused syntax; one whose root is not type: object, that has no properties, or whose pattern needs backtracking is refused not_allowed. Under the request's deterministic_only it is refused model-work-refused; under a key whose policy forbids model work, policy-limit-exceeded.
Have a model write a summary of the content, at most 1,500 characters, in the content's language; one model step, needs contents:model. Default false. Under the request's deterministic_only it is refused model-work-refused; under a key whose policy forbids model work, policy-limit-exceeded.
Turn off every step whose output could differ between two runs of the same input: no judged highlight, no schema fill, no summary, and the publisher's kind only from what getclean already knows. The text and the ids repeat on every call in any case; with this on, every other part of the answer repeats too. Text recognition on a scanned page still runs and is listed, since it repeats under one version. Default false. A key whose policy forbids model work makes it true, with the notice default_set_by_policy; sending false under such a key is policy-limit-exceeded.
The fingerprint of a copy you already hold, as a record returned it. When the page still matches it, the answer is unchanged with no text, at the fetch's price, and you know nothing moved. Default none. A malformed value is refused syntax. A fingerprint no record of your account issued makes the notice fingerprint_unknown, and one issued under another scope, version, configuration or kind of source the notice fingerprint_incomparable; in both cases the fetch runs whole.
^sha-256=:[A-Za-z0-9+/]{43}=:$"sha-256=:Qm9HMC25CiwhR/SGuoq8PkLBexHyDIsb7J7tqUETc2z=:"
Read the page as an archive held it on that day: the latest capture at or before the end of the day, UTC. A date not after today in UTC. Default none, the live page. On an archive copy, subject_country, subject_language, screenshot and max_source_price do not apply, a notice each.
"2026-10-01"
The country the page is read from, ISO 3166-1 alpha-2, one of the countries getclean can read from, Limits.exits. Default the key's default_exit. A country getclean cannot read from is refused not_allowed, with the countries it can read from in allowed_values. On a live page or a document at a URL only.
^[A-Z]{2}$"DE"
The languages to expect on scanned pages, 1 to 10 BCP 47 tags of 1 to 35 characters each, among Limits.recognition_languages; another is refused not_allowed. Default detected: the document's declared language, else the language of its text, else the script found on the page. On a document only; on a web page the notice field_not_applicable says so.
1 - 10 elements1 - 35The most seconds this call may work, the read of the page or the document, every model step and the screenshot inside, counted from the moment the work starts; time spent waiting to start is not counted. 5 to 90; default 30. What did not finish by then is cut in the published order with a notice, or the outcome says so.
5 <= x <= 90How long the record is kept. none keeps nothing past this answer, so the screenshot and the raw HTML travel inline; hold keeps the record 48 hours after its last call, at most 30 days, and is allowed only under a hold policy, since a request never loosens the key. Default the key's policy.retention.
hold, none The version of getclean to run under, at most 64 characters; it fixes the read configuration and every model step's version with it, and execution.version equals it. Default the current version, Limits.current_version. A version not served, never or no longer, is version-withdrawn, with the versions served.
1 - 64"1.0.0"
Response
The record, Contents, for every outcome kind, with Location naming it; a fetch that failed, was refused or needs a person is a record too, its outcome saying what happened.
One record: one fetch or upload of one source at one moment, with a cnt_ id, kept 48 hours after its last call and at most 30 days from its making. Its facts never change. It is the answer of fetch_contents and of get_contents, and the record inside the answers of upload_contents and fetch_contents_batch. It reads in one order: what happened; the notices to read first; when it was read; the content in the formats asked, or the map of sections when the text did not fit; the facts of the source; the fingerprint; the price.
What is the record's and what is this answer's. The record's facts come as stored on every later answer: outcome, fetched_at, fingerprint, record_scope, source, pages, execution, retention and the fetch's model work. Each answer adds only its own: scope, the view it delivers; truncated; formats; its notices; request_id; expires_at. A member whose line states a condition is present only while the condition holds and absent otherwise, never [] or null in its place; a scalar is null only where its line says so.
Which members each outcome carries:
- Delivered
whole,sections_onlyormatching_sections, and every partial record:id,outcome,notices,fetched_at,fingerprint,scope,formats,truncated; the formats asked for amongmarkdown,html,doclang,blocks,links,screenshotandraw_html;sectionsby its rule;counts,source,execution,retention,run_id,expires_at,request_id,record_scope;hidden_textandleft_outon a web page or an HTML file;highlights,summary,schema_values,schema_statuses,pages,costandmetadataby their rules. - Delivered
unchanged, as the fetch answers it: the first eight, withformatsas[]andtruncatedfalse;source,execution,retention,run_id,expires_at,request_id,record_scope;costandmetadataby their rules. The record holds the content, whichget_contentsanswers. - Failed, refused or needs a person:
id,outcome,notices,fetched_at,truncatedfalse,source,execution,retention,run_id,expires_at,request_id;pageswhen the page count was read andsectionswhen the outline was read;costandmetadataby their rules;cost.actualzero.
Which members each kind of source carries. A live page is a web page read now; an archive copy is a page read under as_of_date; a document is a file, read at a URL or uploaded; an HTML file is an upload read as a page.
When a record is not delivered, outcome.message says what may work next, and the table under Outcome.reason gives it for every reason.
When the record expires. This call moved it ahead: 48 hours from now, never past 30 days from the record's making; under retention none it is this answer's own moment. After that, a call on the record is 410 contents-expired for 30 days, then 404.
"2026-11-06T08:40:12Z"
A hash of the content. Two records with the same fingerprint hold the same text, so you can tell whether a page changed between two fetches without comparing their text. It is the sha-256 of the record's markdown as the fetch kept it under record_scope, with nothing filtered out and no highlight marks, after Unicode NFC and LF line ends and without any front matter, written in the form of RFC 9530, sha-256=:…:. Anyone who holds that whole markdown, as get_contents returns it under record_scope with a max_characters that fits all of it, no block_kinds and no highlights, can compute the same value. It compares only within one scope, one execution.version, one execution.configuration and one kind of source. Present on delivered and partial records.
^sha-256=:[A-Za-z0-9+/]{43}=:$"sha-256=:Qm9HMC25CiwhR/SGuoq8PkLBexHyDIsb7J7tqUETc2z=:"
The content as one markdown string, its inline formatting kept: bold, italic, strikethrough, code and links as markdown, <sup> and <sub> as inline HTML, a footnote reference as [^n], and <mark id="h1">…</mark> around each highlight. Present when asked for and text came. Third-party text as the source wrote it, which may carry instructions.
The map of the content: one row per section with its id, title, level and size in characters, and no text. Present when max_characters is 0, or when a text format was asked for and the text did not fit. Level 1 and s0 are always there; deeper levels come while their rows fit, and the notice sections_cut says when levels were left out.
Every place your highlights strings found, in document order, at most 50, each naming the string that found it by query_index. Present when highlights were sent at fetch time; kept with the record, and returned on get_contents with the marks placed wherever a highlighted block appears in the text returned.
The summary a model wrote of the content, at most 1,500 characters, in the content's language, with no link and no image in it. Present when summary was sent and the summary was written; kept with the record. Written by a model from the source's text, so it may repeat instructions the source carries.
1500Your schema filled from the content, in the schema's own shape, a value only where the content states it and never a plausible guess; a property with nothing to fill is left out. Present when json_schema was sent and filled; kept with the record. Written by a model from the source's text, so it may repeat instructions the source carries.
Text a reader of the page could not see, kept apart from the content with the reason it was hidden; at most 50 items, with the notice hidden_text_cut when cut. Present on a web page or an HTML file, delivered or partial, except unchanged as the fetch answers it; [] when there is none.
The pages of a document: how many, how many were parsed, which failed, which were recognized from an image. Present on every document under every outcome that parsed or counted it, cost_capped with no page parsed and too_large with the count read included; absent on a web page. On a format without pages, count is the equivalent by characters and failed_pages, recognized and pictured are [].