Dreaming
The background self-curation that keeps Clarity's memory concise, connected, and current over time.
A growing memory, left alone, gets worse: duplicates accumulate, stale facts pile up, and the connections between stories fade. Dreaming is the background work that keeps Clarity's memory concise, connected, and current by periodically revisiting what it already knows and improving it.
Like sleep consolidating the day, dreaming runs on a schedule, outside ingestion, reworking the graph while no document is waiting. Recording new evidence stays fast and consistent because the heavier, slower curation happens here instead of on the write path. Each pass improves the live graph and then asks for a durable backup, so an improvement is never lost.
What a pass does
Merge duplicates. The same entity is often written two ways by different sources ("Apple" and "Apple Inc.", "Fed" and "Federal Reserve"). It is fused here, where there is time to do it carefully. When two entities are joined, everything attached to the duplicate is carried onto the survivor: its evidence, its relationships, and its story memberships are repointed and de-duplicated, the two sides' corroboration is added together instead of left as parallel copies, and the duplicate is recorded as merged so the consolidation stays auditable. Coverage is deepened, never lost. A guard stops two genuinely different things, such as two distinct stocks, from ever being collapsed together. A weak match is left alone rather than risked.
Re-rank salience. Clarity recomputes how central each entity is to the graph, measuring how connected and how depended-upon it is, so later answers can lead with what matters without re-deriving it on every question.
Regroup stories and themes. Related entities and developments are regrouped into the stories they belong to and rolled up into broader themes, refreshing the connective tissue that lets an answer bridge from one arc to a related one.
Refresh meaning-based search. Entities added without a meaning fingerprint get one filled in, in bounded batches, so meaning-based retrieval keeps working as the graph grows.
Forgetting low-salience detail
A finite memory has to let go. On the same scheduled passes, Clarity compresses stale specifics into a story's running summary, retires superseded facts once nothing still needs them, and archives quiet stories to a compact gist with their history intact. High-value and durable reference facts are kept, and every removal is recorded, so forgetting stays auditable.
Running in the background
Keeping dreaming off the write path is deliberate. A surge of incoming evidence is recorded quickly and stays consistent while curation catches up behind it. The work that makes the graph smart never competes with the work that makes it current. Background passes use the same ordered write lane as ingestion for the graph they touch, so curation never races with recording.
Dreaming only ever rewrites what Clarity already knows by merging, regrouping, re-ranking, and retiring detail. It never invents facts. New knowledge enters only through ingestion and extraction.
For these maintenance steps in the context of the full write path, see Pipeline → Maintenance.
Last updated