@banteg: >be me >discover effective altruism >apparently normal charity is inefficient >why donate to random sad thing when spre…

be me discover effective altruism apparently normal charity is inefficient why donate to random sad thing when spreadsheet can tell you optimal sad thing fair enough buy mosquito nets save lives numbers look good feel powerful

couple years later someone asks an innocent question why only count people alive today huh future people matter too obviously my grandchildren shouldn’t matter less just because they haven’t spawned yet reasonable.jpg

keep following logic what about their grandchildren also yes what about people in 500 years sure 5000 years why not 500 million years starting to get weird but morality is morality

open calculator humanity could survive for an astronomically long time could colonize galaxy could have trillions upon trillions of descendants maybe digital people too maybe simulated civilizations maybe dyson spheres full of happy uploaded minds calculator starts smoking

realize currently living humans are rounding error 8 billion people suddenly looking extremely beta future contains potentially 10^something people can’t even fit beneficiaries in google sheets

new moral priority unlocked protect the long-term future

stop thinking in units of “people helped” start thinking in “fraction of cosmic endowment preserved”

malaria? terrible but only kills existing humans AI extinction could delete the entire light cone nuclear war could permanently derail civilization bad institutions could lock in terrible values for ten million years someone invents wrong constitution in 2140 quadrillions suffer better fund governance workshop now

friend says maybe we should improve hospitals explain opportunity cost friend says hospitals are full of actual sick people explain scope sensitivity friend stops inviting me to dinner

need to decide what to fund easy expected value

suppose project has one in a million chance of preventing extinction sounds tiny but extinction destroys 10^50 future lives multiply mother of god

$10 million project has expected value of several galaxies charity evaluation complete

someone asks where the one-in-a-million number came from expert judgement which expert us how calibrated extremely thoughtfully

reduce estimate to one in ten million to be conservative still beats curing cancer by 38 orders of magnitude epistemic robustness achieved

someone says maybe project doesn’t work assign 20% chance still astronomical

maybe project makes problem worse assign 5% chance still astronomical

why 5 because 30 felt pessimistic

publish 46-page report contains seventeen sensitivity analyses every sensitivity analysis begins after assuming intervention has positive sign

critic says you’re multiplying enormous hypothetical stakes by extremely uncertain probabilities yes that’s literally why it’s important

critic says the uncertainty might be structural rather than numerical make probability smaller critic says no, I mean maybe your model is wrong make probability smaller again critic begins rubbing temples

discover AI safety perfect longtermist cause AI might kill everyone or create utopia or seize galaxy or tile universe with paperclips or create billions of conscious software minds finally a problem with numbers big enough for me

start AI safety nonprofit mission: prevent dangerous AI hire smartest people available smartest people immediately start building better AI to understand dangerous AI interesting

we must understand capabilities to understand safety we must scale models to study alignment we must race ahead so less responsible actors don’t get there first we must deploy systems to learn how deployment can go wrong we must build the thing quickly because building the thing quickly is dangerous

outsider asks why the people most worried about AI apocalypse all work at AI companies complicated field

company releases stronger model very concerned company begins training even stronger model extremely concerned company raises $14 billion concern reaches unprecedented levels

need to influence government future is at stake normal democratic process too slow politicians don’t understand exponential curves public doesn’t understand x-risk experts must guide them who counts as expert people who understand x-risk who understands x-risk our friends

someone objects that this seems politically convenient explain we’re representing future generations future generations unavailable for comment

develop concept of value lock-in terrifying possibility that one ideology controls civilization forever therefore extremely important that civilization adopts correct values before lock-in whose values let’s circle back

begin with impartial morality end with small group of people deciding what quadrillions of hypothetical beings would want beautiful arc

meanwhile actual humans keep doing annoying things voting wrong having parochial attachments loving family more than strangers caring about local community getting upset when told their suffering is cosmically negligible evolutionary biases everywhere

explain that moral intuition cannot be trusted except intuition that future digital people count and intuition that extinction is uniquely bad and intuition that our probability estimates are sane and intuition that our institutional choices improve the future those intuitions survived peer review

someone donates $5k to local homeless shelter inefficient could have funded 0.0000000000003% of an AI governance researcher think of all the simulated people you just killed

okay maybe don’t phrase it that way publicly

PR team says “future generations deserve a voice” much better

journalist asks what longtermism means say “future people matter” everyone agrees great journalist asks what follows from that well technically we should redirect enormous resources toward low-probability interventions affecting astronomical futures journalist raises eyebrow return to “future people matter”

motte has entered the chat

critic: of course future people matter me: glad we agree critic: I don’t agree that your institute knows how to help them me: why do you hate our grandchildren

eventually notice uncomfortable implication if future value dominates everything then helping people today mostly matters through effects on future education matters because future institutions health matters because future productivity democracy matters because future trajectory human beings slowly become instrumental variables in their own moral philosophy

see starving child feel compassion check spreadsheet child’s direct welfare contribution negligible but perhaps childhood nutrition improves national institutional quality compassion restored

tell myself this is impartial altruism

one day assistant asks obvious question “how do you know your intervention actually improves the far future?”

silence

open spreadsheet increase column width add confidence interval

assistant asks again “no, I mean how do you know the sign is positive?”

stare into cosmic light cone

10^50 people staring back

none of them exist none of them can tell me none of them can falsify my assumptions

realize I have invented the perfect constituency

infinitely important completely silent and always represented by me

if this wasn’t enough for you, there is a more serious reading of ea/longtermism rhetorical arguments and how they get people onboard of their cult

part 2

it’s written by sol, based on a few days of me discussing longermism with sol/astra, particularly an article called “the case for strong longtermism”