Where these ideas come from

← Conferral by Design

Almost everything argued on this site was argued first by somebody else, mostly between 1970 and 2024, mostly by people who were not thinking about AI products at all.

That is worth saying plainly and near the top, for two reasons. A framework that does not cite its ancestors reads as reinvention to anyone who knows the field, and rightly. And more usefully: the ancestors are better than the framework. If you read one thing from this page instead of anything else here, you will come out ahead.

So this is the map. Eighteen works, what each established, and what each left open. The last section is the narrow part that is left over once they are all accounted for, which is smaller than it looked before the map was made.

A note on what this page is not. It is not a literature review, it is not complete, and it is weighted toward what a product decision actually touches. The second-ring reading — behavioural economics, science and technology studies, the persuasive-technology critique, the AI governance frameworks — is real and is not here.

Trust, reliance, and the difference between them

The single most consequential thing in this literature is a distinction the industry routinely collapses: trust is an attitude, reliance is a behaviour, and neither is the goal. The goal is reliance calibrated to what the system can actually do.

Lee & See, 2004 · Trust in Automation: Designing for Appropriate Reliance
Human Factors 46(1), 50–80

EstablishesThe target is appropriate reliance, not maximum reliance. Trust should track the system’s real capability; the design problem is calibration, and both over-trust and under-trust are failures.

Leaves openHow a product should behave when its own capability varies by task, and what a team should watch to notice miscalibration early.

Parasuraman & Riley, 1997 · Humans and Automation: Use, Misuse, Disuse, Abuse
Human Factors 39(2), 230–253

EstablishesFour failure modes, not one. Misuse is over-reliance; disuse is rejecting automation that would have helped; abuse is deploying it in a role it should never have had.

Leaves openWhether an engagement number can distinguish any of the four, which it generally cannot.

Mayer, Davis & Schoorman, 1995 · An Integrative Model of Organizational Trust
Academy of Management Review 20(3), 709–734

EstablishesTrust decomposes into perceived ability, benevolence and integrity, and is separable from the risk-taking behaviour it produces.

Leaves openHow any of it is observable from product telemetry, which is where a design rubric has to operate.

Parasuraman, Sheridan & Wickens, 2000 · A Model for Types and Levels of Human Interaction with Automation
IEEE Trans. SMC-A 30(3), 286–297

EstablishesAutomation is not one dial. It applies at different stages — acquiring information, analysing it, deciding, acting — and can sit at a different level in each.

Leaves openHow a permission model should be versioned and revoked as a product’s capability changes underneath it.

Why the numbers go wrong

Two papers explain most of what happens to a product metric under pressure, and one of them predates Goodhart’s law as it is usually cited.

Campbell, 1979 · Assessing the Impact of Planned Social Change
Evaluation and Program Planning 2(1), 67–90

EstablishesThe more a quantitative indicator is used for consequential decisions, the more it distorts the process it was meant to monitor. Pressure gathers around the number and the activity adapts to the measure.

Leaves openWhat to do about it. Naming the corruption does not tell a team which measure to adopt instead, or what they should be willing to let fall.

Cronbach & Meehl, 1955 · Construct Validity in Psychological Tests
Psychological Bulletin 52(4), 281–302

EstablishesA measure’s validity is a claim about what it measures, and it has to be argued and tested. Consistency is not validity. Two people agreeing does not establish that the thing they agree about is real.

Leaves openNothing, really — and this one is why the scorecard here says its validity is not established. It is the standard this work has not yet met.

Leaving, and what a market does when quality is hard to see

Four economics papers do most of the work behind the idea that withdrawal is information and that publishing a number is only sometimes a signal.

Hirschman, 1970 · Exit, Voice, and Loyalty
Harvard University Press

EstablishesWhen quality declines, people can leave or complain, and loyalty shapes which. Exit is not a clean readout of trust: it depends on alternatives, on how costly leaving is, and on whether complaining still works.

Leaves openHow to instrument any of it. But it is the reason a low withdrawal rate can mean lock-in rather than satisfaction, and the reason a rising one can mean the controls finally work.

Akerlof, 1970 · The Market for ‘Lemons’
Quarterly Journal of Economics 84(3), 488–500

EstablishesWhen buyers cannot tell good from bad before purchase, the average price falls to reflect the risk, and good sellers are driven out. Quality uncertainty is a market failure, not a marketing problem.

Leaves openWhat a seller can do about it, which is the next paper.

Spence, 1973 · Job Market Signaling
Quarterly Journal of Economics 87(3), 355–374

EstablishesA signal separates good from bad only if it is more expensive for the worse type to send. Cheap talk conveys nothing precisely because it is cheap.

Leaves openWhether any particular disclosure is actually costly. A self-defined metric, published once, with a definition that can move, is close to cheap talk — which is a correction this site had to make about its own argument.

Morgan & Hunt, 1994 · The Commitment-Trust Theory of Relationship Marketing
Journal of Marketing 58(3), 20–38

EstablishesTrust and commitment are the mediators of durable exchange. The commercial value of trust in a continuing relationship is long-established and does not need re-arguing.

Leaves openAnything about products that act on a customer’s behalf, or about telemetry.

Eisenhardt, 1989 · Agency Theory: An Assessment and Review
Academy of Management Review 14(1), 57–74

EstablishesWhen one party acts for another with different information and interests, the contract has to do work that goodwill will not: observable outcomes, monitoring, and aligned incentives.

Leaves openThe specific clauses. But it is the reason the buy-side instrument ends in contract language rather than principles.

The product layer

Four works that are closest to the actual design decisions, and the only one here with direct experimental evidence about the behaviour this site calls sycophancy.

Amershi et al., 2019 · Guidelines for Human–AI Interaction
CHI 2019, Paper 3

EstablishesEighteen validated design guidelines covering what the system should make clear about its capability, how it should behave when wrong, and how the user should be able to correct and control it.

Leaves openWhat a team should measure to know whether they are following them, and what to do when a guideline costs engagement.

Stray et al., 2024 · Building Human Values into Recommender Systems
ACM Trans. Recommender Systems 2(3), Art. 20

EstablishesValues-driven measurement, reflective versus revealed preference, long-term outcomes, user control, and the causal-inference problems all of it runs into. Much of what a “better metric’ argument wants has been worked through here.

Leaves openThe build/buy/publish decisions a product organisation actually makes, and the case for a specific pair of published numbers.

Sharma et al., 2024 · Towards Understanding Sycophancy in Language Models
ICLR 2024

EstablishesSycophancy is measurable, present across assistants, and traceable to preference data in which human raters reward agreement. It is an incentive artefact, not a quirk.

Leaves openWhether sycophancy damages a person’s ability to rely well over time. That is the claim this site makes and cannot yet evidence.

Friestad & Wright, 1994 · The Persuasion Knowledge Model
Journal of Consumer Research 21(1), 1–31

EstablishesPeople develop knowledge about how they are being persuaded and change their response once they recognise an attempt. Recognition, not disclosure, is what changes behaviour.

Leaves openWhat happens when the persuasion arrives inside an answer the person asked for, which is the case the advertising essay is about.

Disclosure, and the machine layer

Three works that cover ground the “agents will confer on each other” argument sometimes talks as though nobody has occupied.

Bommasani et al., 2023–2025 · The Foundation Model Transparency Index
Stanford CRFM

EstablishesA repeatable public-disclosure methodology: indicators fixed in advance, sources cited, scores published, developers given right of reply, and the whole thing re-run to show movement.

Leaves openDesign incentives. It scores what is disclosed, not whether a product is built so reliance is earned — but methodologically it is the model any disclosure work here should be held to.

NIST SP 800-207, 2020 · Zero Trust Architecture
Rose, Borchert, Mitchell & Connelly

EstablishesNever trust by network position; verify identity per request, grant least privilege, and re-evaluate continuously. The authorisation architecture for machine-to-machine access is already specified.

Leaves openWhether the thing being authorised is worth relying on. Zero trust settles access, not judgement.

Pinyol & Sabater-Mir, 2013 · Computational Trust and Reputation Models for Open Multi-Agent Systems
Artificial Intelligence Review 40(1), 1–25

EstablishesTwo decades of models for how agents decide whether to rely on other agents — direct experience, witness reputation, and their known attacks, including collusion and identity churn.

Leaves openHow any of it interacts with a commercial product’s incentives, which is the part worth writing about.

What is left

After the map, the honest inventory of what this work adds is short. It is worth writing down, because a framework that cannot say what is new about it usually has nothing new about it.

One. The integration. Appropriate reliance comes from human factors, metric corruption from evaluation research, exit from economics, calibration guidelines from HCI, disclosure methodology from transparency indexes, and authorisation from security. Nobody has much reason to put them in one place, because they are separate fields with separate conferences. A product lead deciding what to ship on Tuesday needs them in one place.

Two. The instrument, at the level of a product decision. Not a theory and not a checklist of properties: ten criteria with published anchors, an evidence tag on every judgement, a buy-side version that ends in contract language, and a launch gate that forces the trade into the room before the result arrives. Whether it produces consistent judgements in other people’s hands is not yet established, and whether it measures what it is named for is a further question that consistency would not answer.

Three. Two proposed measures, offered for criticism. A session-origin distribution and a withdrawal rate, specified in enough detail to implement or refuse. The first version of one of them was defeated by a counter-case within hours of publication and has been retired and replaced in public. Neither has been computed on a real product. They are proposals.

Four. A teaching set. Four recurring failure patterns — agreement, interruption, placement, overreach — chosen because they recur and are recognisable, not because they exhaust the space. Anyone claiming a complete taxonomy of ways an AI product can fail its users is overclaiming.

That is the list. It does not include a new theory of trust, a validated measure, a discovery that engagement metrics can mislead, or a claim that anyone else missed something obvious. Those would all be larger claims and none of them would survive this page.

If you are only going to read three

Lee & See (2004) if you build the product — it will change what you think you are optimising for. Hirschman (1970) if you are trying to read your own churn and withdrawal numbers, because it explains why they do not mean what they appear to. Cronbach & Meehl (1955) if anyone has ever handed you a score, including one from here.

This page exists because the alternative — presenting a framework as though the question were new — is both wrong and less useful than the reading it would be hiding.

The methodology these ideas were assembled into. Ten criteria with published anchors, the evidence tags, and the pattern library are at Conferral by Design, with the self-scoring version at the scorecard. Both are free and the definitions are never paywalled.

The essays: Sycophancy is a counterfeit deposit · Session time is an anti-metric for agents · Machine trust will be allocated by contest · An ad in a list is not an ad in an answer · The same task, two designs (runnable) · The two numbers nobody publishes