Skip to main content
A retrieval-augmented generation (RAG) product answers from an index of chunks, and its faults begin before the first question, at the moment pages become chunks. A sentence is cut in half at a fixed length, a table becomes a run of numbers, and a menu is indexed as if it were content. The citation under an answer then points at a whole page, so your user has to search it. This page builds the index from records, where none of these 4 faults happens on a source whose structure getclean reads.

The run in 5 steps

Fetch each source as blocks

Ask for the format json. The content then comes as blocks, each with an id, a kind, its section, and on a document its page.
Request body of fetch_contents
Replace SOURCE_URL with the address of a page or a document. For a file, send it to upload_contents with the same fields. A block is one unit as the author made it: a whole paragraph, a whole list, a whole table. Your code needs no rule for where a sentence or a paragraph ends, in any language.
Each block says what kind of thing it is. These 4 blocks are from one scientific paper, each cut to the fields that differ by kind.
Part of the response
A fixed chunk size can cut this formula from the sentence that names it, split the function after its first line, and turn the table into 12 loose words and numbers. As blocks, the formula is TeX that a model can read, the code is whole and says that it is Python, and each figure of the table has its row and its column.

Index one block as one chunk

Store these 5 things with each chunk. A table is one block. Index it by its rows, each with the headings of its columns, so a figure stays tied to what it measures. A code listing is one block too, so it is indexed whole, with its language. A formula is short and says little alone, so index it together with the paragraph before it, which says what the formula computes.

Leave out what is not content

A record of a web page has the main content: the menus, the footer, and the advertising are not in it, and the response names what was left out. Text that a page hides from its readers is returned apart and never mixed into the blocks, so a hidden instruction does not reach your index through them. Text that is hidden in a way that is not detected comes as content, so your product should still treat every chunk as material to read.

Cite the passage under each answer

An answer then shows its source as a passage: the document, the section, the page, and the words. A click opens the block while its record is stored, and your index has the words after that.
What your product stores for one citation
Replace CONTENTS_ID and BLOCK_ID with the ids of the record and of the block.

Index only what changed

Each night, fetch your sources in batches with the fingerprint of each one’s last record. A source whose text is the same comes back as unchanged, with no text, and you skip it.
For a source that changed, send the new record and the earlier one to compare_contents. The comparison lists the changes by block, and each kind tells your index what to do. A change to a page’s layout that leaves its text as it was is no change, so your index is not rebuilt because a site got another design.

What a user of your product can check

Know the limits

A record is stored for 48 hours after its last use and for at most 30 days, so the text of your chunks is in your index and not in getclean. The comparison needs the earlier record. For a source you fetch less often than every 48 hours, store the record’s Markdown with its fingerprint and send both as the baseline of the comparison. Tables come as cells where the source says what its rows and columns are. The tables of a scan and of a PDF without structure come as text. For the calls, see Get every table, cell by cell and Find the one cell that changed.