the content of a sourceits blocks, each a thing of its owntextthe words, with their bold and their struck texttableevery cell, merged cells keptimageits address and its captionfootnotetied to the sentence it explains
The run in 5 steps
Fetch each source as blocks
Ask for the formatjson. The content then comes as blocks, each with an id, a kind, its section, and on a document its page.
Request body of fetch_contents
SOURCE_URL with the address of a page or a document. For a file, send it to upload_contents with the same fields.
A block is one unit as the author made it: a whole paragraph, a whole list, a whole table. Your code needs no rule for where a sentence or a paragraph ends, in any language.
a paper and its methodF = 2PR / (P + R)its blocks, each with its kindformulaas TeX, exactcodewith its languagetableevery cell, merged cells keptfigureits address and its caption
Part of the response
Index one block as one chunk
Store these 5 things with each chunk.
A table is one block. Index it by its rows, each with the headings of its columns, so a figure stays tied to what it measures.
A code listing is one block too, so it is indexed whole, with its language. A formula is short and says little alone, so index it together with the paragraph before it, which says what the formula computes.
Leave out what is not content
A record of a web page has the main content: the menus, the footer, and the advertising are not in it, and the response names what was left out. Text that a page hides from its readers is returned apart and never mixed into the blocks, so a hidden instruction does not reach your index through them. Text that is hidden in a way that is not detected comes as content, so your product should still treat every chunk as material to read.Cite the passage under each answer
An answer then shows its source as a passage: the document, the section, the page, and the words. A click opens the block while its record is stored, and your index has the words after that.What your product stores for one citation
CONTENTS_ID and BLOCK_ID with the ids of the record and of the block.
Index only what changed
Each night, fetch your sources in batches with the fingerprint of each one’s last record. A source whose text is the same comes back asunchanged, with no text, and you skip it.
the watch list, each nightthe samethe samethe samethe samethe sameone cell: 10% to 8%one new sentencethe page that changed, and only what changedfive pages ask nothing of anybody
compare_contents. The comparison lists the changes by block, and each kind tells your index what to do.
A change to a page’s layout that leaves its text as it was is no change, so your index is not rebuilt because a site got another design.