Evaluation

How Clarity's quality is measured, what blocks a change, and the latest results.

Photo: Stephan Halbach on Unsplash

Clarity is evaluated by running the real agents, with real model calls, over the graph customers read, and grading the answers. Evals are triggered by people. No model grades a card before it is published: publication checks only that every citation resolves to its source.

What is graded

Layer	How	What it catches
Calibrated judge	For each concept on a gold list, whether the answer covers it; for each citation, whether the cited record states the part of the sentence it is attached to	Missing substance, citations that do not back their sentence
Judge	A separate model scores correctness, grounding, causal accuracy, temporal accuracy, completeness, structure, actionability, engagement, plain language and thesis structure (0–100), and must list every material error	Wrong facts, stale facts, inverted causality
Reviewer	Another model splits the answer into assertions, marks each supported, unsupported or contradicted, and gives a yes/no senior-analyst sign-off	Unsupported claims a high score would hide
Output check	Deterministic rejection of internal jargon, "no data" narration and directive buy/sell labels	Anything a client should not see
Operations	Cost per answer, cache ratio, p50/p90 latency, steps, tool failures	Regressions in cost and speed

The judge is never the answer model. A direction inversion caps causal accuracy at 20 and blocks client-readiness. Prose is never graded, rejected or rewritten with regular expressions.

Eval lanes

Lane	Question it answers	Data
Analyst gate	Do the agents answer analyst questions (geopolitics → macro → sector → positioning) well?	Newest nightly production backup, pinned by id
Surface scorecard	Does each surface ship? 16 lanes: 9 artifacts and 7 desks, each rated ships, holds, watch or failed	Each surface's own
Artifact lanes	Is the published market update, company read or portfolio card good? Runs the real producer and grades the output.	Production
Analyst parity	How much of what a person who read every document would find do the agents find? Each miss is traced to extraction, retrieval or synthesis.	Fixed corpora with full source text
Extraction grading	Are entities, links (with direction) and effective dates extracted? Reported as a spread over several runs.	Gold digests
Judge calibration	Is the grader itself right? Authored cases: stale repetition, future leakage, causal inversion, wrong citation.	Fixed cases
Private markets desk	Does underwriting separate what is demonstrated from what is claimed?	Fixed corpus
Personas	Are personalized surfaces suitable for each investor profile?	Persona corpora
Conviction calibration	Were stated convictions right, against the market probability at entry?	Resolved positions

A live run restores the newest complete nightly backup of production into a fresh database and is pinned to that backup's id. Two live runs are comparable only when they share it. A run whose newest backup is more than 48 hours old does not start. Reports list each citation the judge found unsupported beside its sentence.

What blocks a change

Gate	Threshold
Judge overall	≥ 70
Citation grounding	≥ 70%
Concept coverage	≥ 60% where a gold list exists
Every answer	Client-ready: reviewer sign-off, required tools completed, output check passed. One bad answer fails the desk.
Cost and latency	Chat ≤ $0.05 and 30 s p50 · Advisor ≤ $0.05 and 45 s · Market update ≤ $0.10 and 180 s · Company read ≤ $0.15 and 240 s · Portfolio card ≤ $0.15 and 300 s

Judge movement of 3–5 points is noise. Only repeated, larger regressions block. Any change to retrieval, ranking, time handling, prompts, tools or profiles goes through the analyst gate.

Latest results

Date	Lane	Result
22 Sep 2026	Private markets desk (fixed corpus, 5 questions)	Judge 80, citations 100%, sign-off 1 of 5 — does not pass the gate yet. $0.015 per answer, 83% cached, p50 112 s.
17 Sep 2026	Company read (MSFT)	$0.018–0.020 per read, 83–114 s
14 Sep 2026	Company card scorecard	Overall 88, grounding 87, citations 100%
11 Sep 2026	Desk scorecard, production snapshot	Desks 75–83 overall, citations 100%, $0.005–0.016 per answer, 66–81% cached

On 11 September a competing prompt branch regressed several desks by 11–30 points. The noise gate caught it before release.

Run it yourself

The admin panel's evaluation view runs the same agent path over a chosen graph, with a full trace: the exact prompt, every tool call, each rejected and accepted submission, the judge's reasons and a plain-language walkthrough. See In the admin panel.