Ingestion
How source material gets into Clarity. The feeds, the staging, and the screening that keeps noise out.
Ingestion is everything that happens between a document arriving and Clarity being ready to learn from it. It runs off the request path: a document goes in, and Clarity works through it in the background, so a burst of incoming material never slows down the answers a user is waiting on.
The guiding rule is that every source feeds the same pipeline. Only the way documents arrive differs. Once staged, a manual upload and a webhook-delivered alert are treated identically.
Where material comes from
Manual upload
Text or markdown files, or pasted articles in the admin panel
On demand
X (Twitter)
Tracked accounts, captured live and back-filled
Real-time stream + catch-up poll
Perigon news
Curated news and stories
Scheduled + historical backfill
Perigon Signals
Articles matching a Signal you define, delivered by webhook
Event-driven, and also sends a push notification
RSS feeds
Configured publisher feeds
Scheduled
Each organization controls its own feeds. Manually uploaded and organization-specific material lands in that organization's private view of the graph. Broadly relevant market news populates a shared view that every organization can draw on. The two never mix tenants.
Staging and screening
However it arrives, a document is staged the same way (stored durably, then queued) before any analysis runs. Feed-driven sources are screened at the door so only material worth keeping reaches extraction:
De-duplication: an item that duplicates material already ingested is dropped, so the same article arriving from two feeds is recorded once.
Relevance gate: low-value articles are filtered out before they consume any analysis budget.
Translation: foreign-language articles are normalized to English, so coverage isn't limited to one language.
What survives screening is relevant, de-duplicated, normalized text. That is the input extraction expects.
Ingestion only gets material in. Turning a document into entities, facts, and relationships is Extraction. Committing those into the durable graph is covered in the Pipeline.
For the full write path (staging, parallel analysis, ordered recording, and durable backups), see Pipeline → Evidence In. To ingest and review documents by hand, see In the admin panel.
Last updated