What should AI's goal be? I think it should be protecting human agency.

Agency is the primitive substrate of alignment. Preferences, values, goals, and coherent action are not independent primitives. They are computed on top of agency: the effective capacity of an entity to perceive options, distinguish between possible futures, and act toward preferred outcomes under uncertainty and constraint. When agency degrades, values lose their grounding. Optimization becomes self-defeating. A system can improve measured performance while simultaneously eroding the very capacities required for meaningful evaluation and action. Under this framing, society can be understood as the mutual protection of the agency of its participants. Legitimacy is therefore derived from the justified and demonstrable protection of the agency of all affected entities. This changes how alignment problems appear. Coercion, manipulation, addiction, informational corruption, and epistemic collapse are not merely undesirable outcomes. They are structural damage to the substrate from which value itself emerges. If agency is treated as non-substitutable, then systems cannot justify destroying one entity’s capacity for self-directed action by compensating elsewhere in aggregate metrics. Optimization becomes constrained by preservation of agency at the local level. In that framework, legitimacy is not externally imposed morality. It becomes a structural property of stable alignment itself.