“Runner”

The headline

Runner has built the trust product and instrumented the engagement product — it earns delegation better than almost anything in its category and then measures itself with the metrics of the category it just escaped. The permission ladder is genuinely excellent. The dashboard, the notification system, and the failure protocol are all still working for the old economy, and the notification system in particular is actively spending the trust the ladder earns.

Score: 18/36 applicable — capture-financed. The shape is the finding: one category-best 4 sitting on top of five scores of 1 and 2. The trust core is real, and the edges are leaking hard enough to swamp it.

Executive summary

Runner's onboarding is the best permission architecture this reviewer has scored: read-only for week one, drafting unlocked after ten approved suggestions, autonomous sending gated behind an explicit user grant with a one-tap revoke. That is the trust ledger made into UX, and Runner's own data (stated by team) shows users who climb the full ladder churn at one-fifth the rate of users who don't.

Three things are spending what the ladder earns. First, the team's north-star metric is weekly active sessions — for a product whose success state is the user not needing a session. Second, Runner sends an average of 9 notifications per user per week, most of which are re-engagement ("You haven't checked your briefing"), not completion receipts; notification-prompted sessions are 41% of total (observed), and nobody had computed that number before this review. Third, when Runner errs — the double-booked flight class of failure — the user discovers it; there is no self-report path, and the error does not update Runner's subsequent confidence.

The three corrections below are sequenced by leverage. The first costs a dashboard change, the second costs a notification policy, the third costs one sprint. Together they convert Runner from a good product with a capture-financed growth model into the reference case for conferral-designed agents.

Scored criteria

Total: 18/36 applicable (50%). Red-flag override check: flag #5 (failures discovered by users as policy) applies. It changes nothing here, and that is worth stating plainly rather than hiding: the override exists to stop a strong total concealing a structural failure, and this total is concealing nothing. Runner reads capture-financed on the arithmetic alone.

The three highest-leverage corrections

1. Change what the dashboard rewards (criterion 1 + 5's root cause). Replace weekly active sessions as north star with a two-line panel: tasks completed without intervention (should rise) and time-to-left-alone — oversight actions per completed task (should fall). Keep sessions as a diagnostic. Implementation sketch: both metrics are computable from existing logs; the work is a definitions doc and a dashboard, not new telemetry. Expected effect: every downstream decision — notifications, streaks, re-engagement — stops being locally rational.

2. Convert the notification system from capture to receipt (criterion 8, 2). Policy: Runner notifies on completion, anomaly, and permission-boundary events — never on absence. Kill re-engagement sends; kill the streak experiment. Implementation sketch: reclassify the existing notification taxonomy into receipt/anomaly/capture, delete the third class, and report granted share of sessions weekly. Prediction to test: total sessions dip, granted share and ladder progression rise, revocations fall. If that prediction fails, this review is wrong and you should say so publicly — the methodology is testable or it is nothing.

3. Ship the self-report path (criterion 5). For the top three failure modes (scheduling conflicts, wrong-recipient drafts, booking errors): detect → disclose to the user before they find it → state cause → state correction → temporarily lower Runner's own autonomy on that task class until re-earned. Implementation sketch: detection heuristics already exist in the QA pipeline; the missing piece is routing them to the user instead of the internal dashboard. Then instrument reliance retention after errors — the number that will justify the whole sprint.

The metric panel to instrument

Granted share of sessions (now 59%, observed — protect it) · permission-ladder progression and revocation rate, attributed · reliance retention after errors (build after correction 3) · endorsement-on-reflection (sample "glad Runner handled this without asking?" 24h after autonomous actions) · time-to-left-alone (the headline; should fall).

Honest limits

Scores 4 and 6 rest partly on team statements not independently verified; a week of production traces would firm both. This review evaluates design and strategy — it makes no claim about model quality, safety properties, or architecture, which are your engineers' domain and were not examined.

The bridge

This review scored the product as it ships today. What it didn't do is sit with the team while the corrections land — the definitions fights, the notification-policy exceptions, the quarter-two re-score. That is the advisory retainer, and correction 1 is where teams most often need the outside voice in the room. One re-score is included with it.