Skip to main content
The web and its documents were made for people to look at. An agent that does research needs something else from a source than a person does. It needs the content without what surrounds it, in parts it can afford to read, with an address for every passage, and with a statement of what each value rests on. All 4 are what getclean returns, as the layer between an agent and its sources. It has 2 families of calls. The search family finds sources for a question, and the contents family reads one source.

A run has 5 steps

Every research run, in any field, goes through the same 5 steps. A call exists for each.

Find sources

search takes one question in up to 20 wordings, in any language, and runs it on the web and on specialist datasources at once. One ranked list comes back. Each result says who publishes it and whether it is a primary source, so official texts can be put before the commentary on them. The response also says what was searched, what was dropped, and what could not be searched. Start with Ask one question in 20 wordings.

Read a source

fetch_contents takes an address, and upload_contents takes a file. Both return the same thing: the main content of the source, with its headings, its tables, and its footnotes as the source has them, without the menus and the advertising of a web page. The content is stored as a record, and every later call reads that copy. A source that is too long to return comes back as its outline, one row for each section with its size. The agent then reads the 2 sections it needs and not all 300 pages. Start with Fetch what a page says, not what surrounds it.

Find the part that counts

A record can be asked. highlight_contents takes up to 5 phrases or questions and returns the passages that bear on each. fill_schema fills your own fields, each with the words it was taken from, and says of an empty field whether the source does not state it or a part of the source was not read. list_blocks returns the tables cell by cell. Start with Highlight the evidence for each question.

Cite it

Whatever comes back has its place: the record, the section, the block, and on a document the printed page. These places are the same on every read of a record, so a sentence of a report can name the passage it rests on, and anyone can open that passage again. Start with Cite the passage, not the page.

Check the work

Because the record is stored and its places do not move, the check costs less than the work. A second agent, or a few lines of code, sends each sentence of a draft against its source and learns whether the words are there. A fingerprint says whether a source changed after it was read, a comparison says what changed, and a fetch with a date returns the page as it was on that day. Start with Check every sentence of a draft against its source.

A record is the unit of everything

One fetch or one upload makes one record: the copy of one source at one moment. It has an id, a time, and a fingerprint, which is a hash of its text. Under one version and one configuration of getclean, the same source gives the same text and the same places on every fetch. A record is stored for 48 hours after the last call that used it, and for at most 30 days. Reading it again costs nothing. For what a record has and how long it lasts, see Records.

A model works only where you ask

The content, the outline, the places, and the comparison are the same on every read of the same source, and so is a highlight found word for word. A model works in named steps: it writes a summary, fills a schema, and judges which passage bears on a question. In a search, a model ranks the results and chooses datasources unless you turn that off. Every response names each model step that ran, and every value says what it rests on: a highlight is exact, near, or judged, and an empty field is absent or unread. An agent can then quote only what was found word for word and treat a judgment as a passage to read. One switch, deterministic_only, turns every model step off.

The reasoning stays with the agent

The layer does not write the report, and it does not decide what a source is worth. It returns facts about a source, the content, and the evidence, and the agent that calls it reasons from them. Nothing is translated, and no page is paid for or signed in to on your behalf. A page that cannot be read comes back with the reason and with what can work next, at no charge. To make the first 3 calls, see Find, read, and cite a source in 3 calls.