@Miles_Brundage: Thread on "The Future Worth Building is Human" by Thinking Machines Lab, aka Thinking Machines, aka Thinky Overall I fo…

Thread on “The Future Worth Building is Human” by Thinking Machines Lab, aka Thinking Machines, aka Thinky

Overall I found it thoughtful + I’m glad to see competition in the AI company vision market.

Some things that I want to call special attention to / am unsure on…

Before going into detail, here’s the big picture: in many respects Thinky is trying to articulate in detail + operationalize stuff that other frontier AI companies state as vague aspirations but do little to actually ensure (i.e. AI augmenting instead of substituting for humans).

Which is why I say I’m glad to see competition here - AI researchers/engineers have a lot of options, and they should use that power to “reward” companies with good missions/values/track records etc. with their labor, and I hope this raises the bar in the industry a bit.

OK, now getting into more detail…

They start by emphasizing distributed knowledge as a motivation for their approach (along with some nice Polanyi and Hayek citations).

I dig the classical liberal, anti-concentration-of-power vibes…

Simplifying the logic a bit, they essentially say that knowledge and values are distributed all over the place, where humans are, so AI development should also be distributed all over the place in order to stay aligned with those humans.

There’s a bit more to it than that and it’s worth reading in full (will link at the end) but here’s one of my main uncertainties:

It seems to me like an open empirical Q how much fine-tuning specifically (vs. customization more generally including prompting) is needed for…

…value-related customization, and how much fine-tuning specifically (vs. customization more generally including RAG, etc.) is needed for knowledge-related customization.

Maybe the answer is “a lot”/“completely”! I just don’t know / have not thought as carefully about this as I’d like.

I do think that concentration of power in a few companies/governments is a huge concern even if, from an efficiency perspective, prompting goes a long way…

…but is fine tuning the key lever there, vs. checks on those big companies, or competition among them, or (etc.)?

And is cloud-based fine-tuning enough, or do you need straight up local open weight models?

I dunno. I hope the essay sparks discussion!

They make some technical points here very briefly that I think merit longer discussion another day, e.g. they sort of imply centralized models will always be vulnerable to jailbreaks…

Maybe! But also they will still tend to be more hardened than a random fine-tuned model…

…but also a random fine-tuned model will have fewer attackers trying to break it, and… etc.

The equilibrium seems unclear here, and I will be very interested to see how Thinky’s thinking evolves on fine-tuning safety. Expecting more from them in the future here.

Note: they talk more about fine-tuning than some of their other technical bets they are investing in, and some of those other bets still make sense even if one isn’t 100% sold on the fine-tuning story.

I’m v. excited to see more augmentation-motivated UX exploration for example.

Later they briefly recapitulate the Intelligence Curse argument (the authors of which work there now, post acquisition of Workshop Labs).

I think that’s a good/important essay but I will make one point on it, which is relevant to the “other technical bets” thing above.

The paraphrase they use is: “Power that needs nothing from people loses the incentive to care for their needs and values, caring instead for its own preservation.”

That’s a valid concern! But ideally Thinky’s strategy would be robust to a scenario in which humans do become…

…economically redundant eventually, even if Thinky tries to delay that for a while.

I’d love to eventually retire + not have the AIs steal my stuff/kill just bc I’m too dumb to add much economic value.

I think of this as a defense-in-depth thing - delay obsolescence somewhat via augment-y stuff, make AIs value humans having a say, require some inefficient human sand in the gears of the economy, make AIs need your biometric/cryptographic vote on some stuff, and (etc.)

I think Thinky could make a very valuable contribution to technical innovation as well as policy debates on such things.

No need to rely on a single thing (obsolescence delay generally or fine-tuning specifically).

OK gonna stop there… here’s the essay, which also has a banger of a final sentence.

Nice thoughtful response, @Miles_Brundage.

I’d like to get your thoughts on one of the fronts that we’re facing on the more cultural side of the AI progress train: labs pathologizing weirdness and individual difference and labeling it “safety.”

I think this approach risks cultural denuding at population scale and damnit, as someone who remembers when the internet was weird, I think we lose something when everything starts to look and sound the same.

They write:

“Humanity has flourished through individual weirdness and creative tension. We envision alignment as a feature not of a single model but of an ecosystem of AIs raised in different places, disagreeing, competing, and learning from each other. We believe in keeping the weirdness alive.”

Amen.

I wish more labs had this vision, and more individuals had the courage to speak out on behalf of this.