Ingest pipeline

An ingest pipeline fetches source documents, extracts structured fields and writes records. This site's runs daily against regulator indexes, caches everything it fetches, and never re-requests a document it already holds.

data and analysis · updated 2026-09-08

See also

• Technique tag — A technique tag links a case record to a manipulation technique. Tags are assigned by keyword rules over the regulator's own text,…
• Rate limiting — Rate limiting restricts how often a client makes requests. This site's crawler serialises requests per host with a delay and ident…

Terms that refer here

Rate limiting

Back to the full glossary — 261 defined terms.