Skip to main content
A research agent can take a failed fetch for a success: the page of a bot challenge for the article, or an empty page for “nothing to report”. So the first field of every record says what happened at the source, before any content. An agent reads outcome first and the text second.

An outcome has 3 parts

outcome.kind is one of 5 values, and code can branch on it. outcome.reason says which case it is, and outcome.message says in one sentence what can work next, if there is a next step. A fetch that met a bot challenge has this outcome.
Part of a response
The record of such a fetch has no content, so nothing in it can pass for the article.

Each reason says what can work next

A fetch or an upload can end with these reasons, among others. reason is an open list: treat a value you do not know as a new one, and read message.

An outcome says what came, not whether it is the right page

whole means that everything you asked for fit. It does not mean that the page is what its address promises. A wall is recognized by its signs, so an error page that is sent as a normal page, or a wall with no known sign, comes back as content. A page that declares its content as not free has the notice paywall_declared, whatever came.

A search has outcomes too

A search as a whole has an outcome, and so has each datasource in its receipt. Their reasons include these 5.

A fetch that did not work costs nothing

A fetch that failed, was refused, or needs a person is not charged. A partial fetch is charged for the part that came.

Notices come before the content

A notice is a fact that changes how the response should be read. It is not an error, and the call succeeded. notices is a list, and these notices bear on how far to trust the text. Other notices say that something was cut to fit a size limit, such as highlights_cut or total_characters_cut, and each names the size limit. A notice that belongs to a record is stored with it. It comes back first on every later read of that record, marked with held set to true, so a second agent that reads the record learns what the first one was told.