Extraction
How Clarity turns one document into structured entities, facts, and relationships.
Extraction is the step that reads a single document and turns it into a structured proposal of what the document says. This proposal is the raw material of the ontology. It is where unstructured prose becomes entities, the facts about them, and the relationships between them, all in one pass.
Extraction writes nothing to the durable graph. Its output is a proposal, held to one side, reconciled against existing knowledge and committed automatically later in the pipeline.
What gets pulled out
Entities: every specific, named thing the document is about, such as companies, people, places, events, assets, and topics. Each entity carries its aliases (abbreviations, tickers, and acronyms) which become the keywords that let later documents match the same entity, so "AAPL" resolves to Apple and "Fed" to the Federal Reserve.
Claims: every distinct fact, quote, or figure, captured as its own atomic claim with its own time and confidence. Keeping facts atomic lets each one be cited, dated, and superseded independently.
Relationships: the typed links between entities. Anything the document blames an effect on is pulled out as a cause with a causal link, which captures the "why" alongside the "what."
Filtering generic entities
Extraction errs toward inclusion, then filters. A second pass removes generic placeholders ("the market", "analysts", "investors", "the company") that would clutter the graph without naming anything. The rough test is: could you look this up by name and find a specific article? If not, it's dropped. This keeps the graph anchored to real, identifiable things.
Time on every fact
Extraction captures several distinct timestamps per fact, which is what lets Clarity answer both "what is true now?" and "what was true then?":
Event date
when the event happened or was announced
Observed at
when Clarity saw it (the publication date)
Valid from
when the fact starts being true
Valid to
when it stops being true (open if still current)
Two rules keep this honest. A future-dated transition takes its valid from from the effective date rather than the announcement date: "Cook will become executive chairman effective September 1" is stored as valid from September 1, so Clarity does not call him chairman the day the news broke. It waits for the day it actually takes effect. And a cause-and-effect link inherits the document's date even when the sentence stating it carries no date of its own: an article published on May 9 saying higher fuel and fertilizer costs are pressuring food prices anchors that whole causal chain to May 9, so a later "what was driving food prices in May?" finds it. Without that anchor the link would float free of time and drop out of any date-bounded question.
Context versus causal
Every relationship is directional and typed. A structural tie ("appointed CEO of", "supplies") is context. An assertion of cause and effect ("the rate cut drove the rally") is causal, and carries a strength, lag, and direction so the chain can be traversed both ways later. Direction lives on the link, never baked into a name: "oil supply" is the variable, and whether it rose or fell is a property of the relationship, so sentiment is never mistaken for direction.
A causal link is always oriented from the driver to the thing it affects, the cause then the effect, regardless of how the sentence was phrased. "Siri is powered by Apple Intelligence" is captured as Apple Intelligence → powers → Siri rather than the reverse. "shares fell on higher rates" becomes rates → pressure → shares. This orientation makes a chain traversable: when one capability powers another which is in turn delayed by a regulation, Clarity stores three peer links it can walk end to end, rather than collapsing them into "everything belongs to the company." Capturing the real mechanism between two things, and pointing the arrow the right way, is the difference between a graph that can explain why and one that can only list what.
Completing the chain
Reading a long article in one pass, extraction sometimes captures the pieces of a causal chain but leaves a gap in the middle. It records that oil prices rose and that food prices rose, but the cost link in between is left dangling, so the path no longer connects end to end. This is most common where several drivers converge on one outcome (diesel, fertilizer, and tight grain supply all pushing up food prices).
A dedicated second pass then audits the chain. It re-reads the article against the links already captured and adds only the missing connectors needed to make each chain traversable: wiring a dangling intermediate onward to the outcome it drives, and making an implied convergence explicit by linking every contributing driver to the outcome. It only connects entities the first pass already found and only links the article actually supports. It never invents a node or a relationship the text doesn't state. The result is that multi-step causal stories, such as a geopolitical shock reaching consumer prices through energy, freight, and fertilizer, stay connected all the way through, which lets Clarity trace them later.
A single data-rich article typically yields 30–60 entities and 30–50 claims. The Pipeline walks a real article through extraction end to end, with the structured records it produces.
The proposal extraction produces is then reconciled against what Clarity already knows and committed. See Memory and Pipeline → Record.
Last updated