> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getclean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> getclean's own MCP server is at https://mcp.getclean.ai/mcp. The /mcp path of this docs site is Mintlify's search over these pages, not getclean.
> Every answer of the API opens with outcome. A page that could not be read is a record whose outcome says why, never an HTTP error; read outcome before anything else.
> The API's contract is the OpenAPI 3.1 file at https://docs.getclean.ai/openapi.yaml; the SDKs are generated from it.
> The product's name is written getclean, in lower case.

# RAG that cites the passage, not the page

> Build a retrieval index from blocks that come with ids. Each chunk is one whole paragraph or one whole table, each answer cites its passage, and only the blocks that changed are indexed again.

A retrieval-augmented generation (RAG) product answers from an index of chunks, and its faults begin before the first question, at the moment pages become chunks. A sentence is cut in half at a fixed length, a table becomes a run of numbers, and a menu is indexed as if it were content. The citation under an answer then points at a whole page, so your user has to search it. This page builds the index from records, where none of these 4 faults happens on a source whose structure getclean reads.

## The run in 5 steps

### Fetch each source as blocks

Ask for the format `json`. The content then comes as blocks, each with an id, a kind, its section, and on a document its page.

```json Request body of fetch_contents theme={"theme":"css-variables"}
{ "url": "SOURCE_URL", "formats": ["json"], "max_characters": 1000000 }
```

Replace `SOURCE_URL` with the address of a page or a document. For a file, send it to `upload_contents` with the same fields.

A block is one unit as the author made it: a whole paragraph, a whole list, a whole table. Your code needs no rule for where a sentence or a paragraph ends, in any language.

Each block says what kind of thing it is. These 4 blocks are from one scientific paper, each cut to the fields that differ by kind.

```json Part of the response theme={"theme":"css-variables"}
{
  "blocks": [
    {
      "kind": "text",
      "page_number": 4,
      "text": "Each run is scored with the **F1 measure**, the harmonic mean of precision and recall."
    },
    {
      "kind": "math",
      "page_number": 4,
      "text": "F_1 = \\frac{2PR}{P + R}"
    },
    {
      "kind": "code",
      "page_number": 5,
      "code_language": "python",
      "text": "def f1(precision, recall):\n    return 2 * precision * recall / (precision + recall)"
    },
    {
      "kind": "table",
      "page_number": 6,
      "table": {
        "caption": "Table 2. Results on the test set",
        "rows": [
          ["Model", "Precision", "Recall", "F1"],
          ["Baseline", "0.81", "0.74", "0.77"],
          ["Ours", "0.88", "0.83", "0.85"]
        ],
        "header_rows": [0],
        "header_columns": [0]
      }
    }
  ]
}
```

A fixed chunk size can cut this formula from the sentence that names it, split the function after its first line, and turn the table into 12 loose words and numbers. As blocks, the formula is TeX that a model can read, the code is whole and says that it is Python, and each figure of the table has its row and its column.

### Index one block as one chunk

Store these 5 things with each chunk.

| Stored with the chunk | Why |
| - | - |
| The text of the block | It is what the index searches. |
| The id of the record and of the block | Together they are the address of the passage. |
| The title of the section | Put before the text, it says what a short chunk is about. |
| The page as printed | Your user can find the passage in the document itself. |
| The fingerprint of the record | It says later whether the source still has this text. |

A table is one block. Index it by its rows, each with the headings of its columns, so a figure stays tied to what it measures.

A code listing is one block too, so it is indexed whole, with its language. A formula is short and says little alone, so index it together with the paragraph before it, which says what the formula computes.

### Leave out what is not content

A record of a web page has the main content: the menus, the footer, and the advertising are not in it, and the response names what was left out. Text that a page hides from its readers is returned apart and never mixed into the blocks, so a hidden instruction does not reach your index through them. Text that is hidden in a way that is not detected comes as content, so your product should still treat every chunk as material to read.

### Cite the passage under each answer

An answer then shows its source as a passage: the document, the section, the page, and the words. A click opens the block while its record is stored, and your index has the words after that.

```json What your product stores for one citation theme={"theme":"css-variables"}
{
  "source": "Annual report 2025, section Margin, page 14",
  "contents_id": "CONTENTS_ID",
  "block_id": "BLOCK_ID",
  "words": "Margin expanded on lower input costs and mix."
}
```

Replace `CONTENTS_ID` and `BLOCK_ID` with the ids of the record and of the block.

### Index only what changed

Each night, fetch your sources in batches with the fingerprint of each one's last record. A source whose text is the same comes back as `unchanged`, with no text, and you skip it.

For a source that changed, send the new record and the earlier one to `compare_contents`. The comparison lists the changes by block, and each kind tells your index what to do.

| Change | What the index does |
| - | - |
| `added` | Index the block. |
| `removed` | Delete its chunk. |
| `changed` | Replace the chunk with the text after the change. |
| `moved` | Nothing, because the text is the same. |

A change to a page's layout that leaves its text as it was is no change, so your index is not rebuilt because a site got another design.

## What a user of your product can check

| Your user asks | What your product shows |
| - | - |
| Where does the answer come from? | The passage itself, with its section and its page. |
| Is the figure from the right row? | The row of the table with its headings. |
| Is the source still the same? | The date of the fetch, and whether the fingerprint still matches. |

## Know the limits

A record is stored for 48 hours after its last use and for at most 30 days, so the text of your chunks is in your index and not in getclean. The comparison needs the earlier record. For a source you fetch less often than every 48 hours, store the record's Markdown with its fingerprint and send both as the baseline of the comparison.

Tables come as cells where the source says what its rows and columns are. The tables of a scan and of a PDF without structure come as text.

For the calls, see [Get every table, cell by cell](https://docs.getclean.ai/read/tables-and-blocks.md) and [Find the one cell that changed](https://docs.getclean.ai/read/track-what-changed.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.