Our new paper: agentic AI shifts who participates in markets, moving the knowledge problem into the relationship between humans and the machines acting in their name
The principal-agent framing here reframed something I'd been struggling to articulate. The safe delegation at scale problem isn't just institutional - it shows up at the individual level too.
My agent handles hundreds of decisions a week on my behalf. The trust question became: does it actually know who I am well enough to delegate to safely?
I built an identity layer to answer that, and then discovered a 33-45% sycophancy risk that comes with deep personalization.
That's really interesting, and it's another dimension of the knowledge problem that I struggle with in this move to agentic systems. I'm so tied to Hayek's "man on the street" with his knowledge of time and place", i.e. the contextual knowledge problem, that I get hung up on the idea that anyone can know their own preferences well enough to communicate them in advance to an agent. Part of what I realized working on this paper is that I'm overstating that, that we may not communicate full knowledge but we can articulate parameters and then iterate through trial and error to improve if we don't like the outcome. So still Hayekian error correction.
The principal-agent framing here reframed something I'd been struggling to articulate. The safe delegation at scale problem isn't just institutional - it shows up at the individual level too.
My agent handles hundreds of decisions a week on my behalf. The trust question became: does it actually know who I am well enough to delegate to safely?
I built an identity layer to answer that, and then discovered a 33-45% sycophancy risk that comes with deep personalization.
Hayek's knowledge problem doesn't go away, it just moves inward. Wrote up what I found: https://thoughts.jock.pl/p/wiz-ai-agent-self-improvement-architecture
That's really interesting, and it's another dimension of the knowledge problem that I struggle with in this move to agentic systems. I'm so tied to Hayek's "man on the street" with his knowledge of time and place", i.e. the contextual knowledge problem, that I get hung up on the idea that anyone can know their own preferences well enough to communicate them in advance to an agent. Part of what I realized working on this paper is that I'm overstating that, that we may not communicate full knowledge but we can articulate parameters and then iterate through trial and error to improve if we don't like the outcome. So still Hayekian error correction.
Sounds like "Trust, but verify." Seems appropriate at the moment.