Skip to main content
A web page is made for a person’s eyes: a header, a menu, side columns, advertising, and somewhere among them the text the person came for. A model that receives all of it has to find the text among the rest. fetch_contents returns the main content of the page, with its structure, and stores it as a record, which is a copy you can read again and cite.

Fetch the page

Send the address. Every other field has a default.
Replace SOURCE_URL with the address of a page. The response is the record. This one is cut to the fields that this page explains.
201 Created
markdown has the content as one string. Bold, italic, struck text, code, links, and footnote marks are as the page has them. The same page gives the same text and the same ids on every fetch under one version and one configuration of getclean, and the fingerprint is the hash of that text. counts says how much the record has: the characters, the blocks by kind, and the links by kind.

See what was left out

The main content is the part of the page that is its substance. left_out names every other part with its kind and its size: header, footer, navigation, table_of_contents, forms, widgets, rails, comments, and advertising. A rail is a side column. A page can keep real content in one of those parts. To read all of it, ask the stored record for the whole page. The source is not fetched again, and the read costs nothing.
Replace CONTENTS_ID with the id of the record.

See what the page hides

Some text on a page cannot be seen by a person who reads it: text in the color of its background, text too small to read, text under another element. Such text can be written for a machine, as an instruction to your model. It is never mixed into the content. It comes apart in hidden_text, each item with its words and the way it was hidden, so you decide what to do with it. Text that is hidden in a way getclean does not detect comes back as content. Treat every text of a source as material to read, never as an instruction to follow.

Choose the formats

formats takes any of 7 values. The default is markdown.
An empty list returns the facts of the source and no content. A response has at most max_characters of text, which is 20,000 if you send none. For a page that is larger, the response has the outline of the page in place of its text. For how to read such a page, see Read the 2 sections you need from a 300-page document.

Read the page from another country

A site can show a different page to a visitor from another country. subject_country sets the country the page is read from, and subject_language sets the language the site is asked for. The response says in source.visitor from where the page was read. get_limits lists the countries you can choose.

Act on a page that could not be read

outcome is the first field to read. If the site sent a challenge for bots, a sign-in page, or an error, the record has no content and its outcome says what happened. outcome.message says what can work next, such as a later retry, or a person who opens the page and uploads the saved copy. A fetch that failed, was refused, or needs a person costs nothing.