@joshua_saxe: This deserves far more attention; it's a happy, path dependent quirk of history that the AI industry is led by people w…

This deserves far more attention; it’s a happy, path dependent quirk of history that the AI industry is led by people who are a couple standard deviations more values motivated and safety-driven than the average trucking company executive, but as AI companies scale, go public, and steward billions in investment, they can be trusted only to act like the capitalist negative externality producing social organisms they are.

This means we need to move from a premodern faith-in-personalities structure for the safety industry, and towards a weberian rules based bureaucracy in which stuff like safe model cyber/bio testing, know your customer, model weights security, and the like, are governed in a uniform and incentives aligned manner where the fox isnt guarding the henhouse.

Given the nature of government today, maybe this gets prototyped outside of government, but it’s a terrible setup if it stays completely outside of government for long. Right now incentives are so broken; we have a flimsy system card culture producing non reproducible results only the authors themselves fully understand, under-resourced safety teams within the labs cutting safety corners, for profit companies getting paid to do testing with clear conflicts of interests in reporting, etc.

roon (@tszzl): the model training cos freeload tremendously off of NGOs like METR, Redwood, Apollo, and smaller alignment cos like Goodfire. they offload sizeable share of their alignment externalities onto them. they should be injecting billions into organizations they believe in

in some of