bytes 0–3ASCIIThe magic string NFH1.
Example: NFH1
Documents
Each document arrives as a JSON header you can route on, then the enriched document as one zstd frame.
Explanation
Delivers each enriched document as a binary frame: a plain JSON header you can route on, then the document as one zstd frame. Use it to decompress only the documents you keep, since filtering on the header costs you nothing and keeps your CPU use low.
Response
Frame you receive
application/x-nfh1 (JSON header + zstd JSON)These are the fields you can build against. Nested names use dot notation; optional sections are called out in their descriptions.
bytes 4–7uint32 (big-endian)Header length in bytes.
Example: 412
headerobject (UTF-8 JSON)Uncompressed metadata. Read it without decompressing.
Example: {"cursor": "...", "arrived_ms": 1793880000000, "facets": {...}}
remainderzstd frameOne zstd frame holding the document JSON, unchanged from NOSIBLE.
Example: 28 b5 2f fd ...
header.cursorstringOpaque position of this document. Save it to resume.
Example: eyJ2IjoxLC4uLn0
header.arrived_msintegerArrival time at the stream, Unix milliseconds. Not publication time.
Example: 1793880000000
header.time_msintegerWhen the server prepared this message, UTC Unix milliseconds.
Example: 1793880000100
header.facetsobjectSelected document fields, flat and uncompressed, so you can route without decompressing. Names match the document fields and values keep their original casing and structure. Facets always describe the whole document, even when fields trims the body.
Example: {"hash": "...", "netloc": "news.example.com", ...}
facets.hashstringThe document's request.hash. Deduplicate on it with visited_timestamp.
Example: 9f2c1e...
facets.netlocstringThe document's request.netloc.
Example: news.example.com
facets.published_timestampstringThe document's page.published_timestamp.
Example: 2026-11-05T11:42:00Z
facets.visited_timestampstringCrawl time (page.visited_timestamp), RFC 3339.
Example: 2026-11-05T12:00:00Z
facets.languagesobjectLanguage codes and scores.
Example: {"en": 0.98}
facets.entitiesobjectEntity rows grouped by label, as in the document.
Example: {"ORG": [{"name": "Shell", "mention_count": 4, "confidence": 0.97}]}
facets.locationsobject[]Resolved geographic records, as in the document.
Example: [{"country": "France", "region": "Western Europe"}]
facets.ontologiesobject[]Classifications with their hierarchy fields and probability, as in the document.
Example: [{"ontology": "gics", "sector": "Energy", "probability": 0.91}]
facets.tickersobject[]Mapped listed companies, as in the document.
Example: [{"ticker": "SHEL.L", "score": 0.93}]
facets.has_tickersbooleanTrue when a ticker was mapped.
Example: true
document.requestobjectThe crawl request, including url and hash. Always present.
Example: {"url": "https://news.example.com/...", "hash": "9f2c1e..."}
document.pageobjectTitle, description, timestamps and embeddings. visited_timestamp is always present.
Example: {"title": "...", "visited_timestamp": "2026-11-05T12:00:00Z"}
document.page.embeddingsobject768-dimension vector under embeddinggemma-2: base64 of little-endian int16, each value round(v × 10000).
Example: {"embeddinggemma-2": "<2048 base64 characters>"}
document.languagesobjectDetected languages and their share of the text.
Example: {"en": 0.98}
document.full_textstringExtracted page text.
Example: Shell announced...
document.snippetsobjectPassages of the text; each may carry its own embeddings.
Example: {...}
document.statisticsobjectCounts describing the page.
Example: {...}
document.entitiesobjectEntities by label (PERSON, ORG, GPE, LOC, PRODUCT) as {name, mention_count, confidence}.
Example: {"ORG": [{"name": "Shell", "mention_count": 4, "confidence": 0.97}]}
document.tickersobject[]Mapped listed companies: ticker, score, why, companyv5_id and more.
Example: [{"ticker": "SHEL.L", "score": 0.93, "companyv5_id": "c5-..."}]
document.ontologiesobject[]Classifications, most probable first: ontology, hierarchy fields and probability. Each family contributes its top three leaf predictions with an original probability of at least 0.5.
Example: [{"ontology": "gics", "sector": "Energy", "probability": 0.91}]
document.locationsobject[]Geocoded places: continent, region, country, lat, lng, source, matched_on.
Example: [{"country": "France", "lat": 46.2, "lng": 2.2, "matched_on": "Paris"}]
document.has_tickersbooleanTrue when tickers is not empty.
Example: true
document.signalsobjectOptional. financial_sentiment and forward_looking classifications, each with a label and probabilities. Present only when both classifiers succeed, so accept documents with or without it.
Example: {"financial_sentiment": {"label": "..."}, "forward_looking": {"label": "..."}}
Errors and close codes
WebSocketBINARYA binary frame. The response example shows its header and decoded document.
1000
slow_consumer — Your client did not read frames within 30 seconds. Reconnect with your cursor.1000
superseded — Another socket opened the same subscription name with the same key. Use a distinct name for each independent connection.1001
server_shutdown — The server is restarting. Reconnect with your cursor.1011
internal — Unexpected server error. Reconnect with your cursor.