Verifiable Domains Will Eat The World

Way back in the Before Times, before social media and the iPhone — even before the towers fell and the modern security state was stood up in their place — we online types would amuse one another by making fun of PowerPoint presentations. PowerPoint was the Nickelback of office software, and performatively hating on it was part of a culture of office humor that now seems to have gone the way of both offices and humor.

Probably the most famous PowerPoint spoof of that era was Peter Norvig’s slide deck rendition of Lincoln’s Gettysburg address.

Every word of Lincoln’s two-hundred seventy-two word oration fits so precisely with every other word, that despite its elevated register there’s a naturalness and wholeness to it that makes it feel like it could only ever have come into being in exactly this form.

Norvig’s PowerPoint (per)version of the speech was unexpected and funny because we have a vague but deep sense that the compelling particulars of that speech’s construction carry a kind of truth that a dry recounting of the same bare facts just can’t carry.

There’s information, and then there’s truth. The two are clearly related, but somehow not actually identical. And in that gap between “information” and “truth” is a whole family of ancient and modern disciplines with names like “rhetoric” and “aesthetics” and “hermeneutics”.

Load-bearing complications

I hate to dwell on this “form vs. substance” issue much more, because I think you get the point, but I want to pull one more thread before I turn to the topic of the controversy surrounding Anthropic’s watermarking, and to the AI nerd meta-controversy over the normie controversy surrounding that same issue.

A few weeks ago, before the Arday affair and the Hugging Face hack — even before Nolan’s Odyssey was released — we all endured a full Discourse Cycle on the politics of Emily Wilson’s Homer translations.

I point your attention back to the much-discussed opening lines of Wilson’s translation of The Odyssey:

Tell me about a complicated man. Muse, tell me how he wandered and was lost when he had wrecked the holy town of Troy, and where he went, and who he met, the pain he suffered in the storms at sea, and how he worked to save his life and bring his men back home.

One of the words Wilson’s critics focused on was πολύτροπος (polytropos), which she translated as “complicated.” It has a range of meaning indicated by its constituent parts — “poly” for “many”, and “tropos” for “way” or “turn”. Lattimore’s classic translation renders it “many ways,” which conveys the sense that Odysseus did a lot of wandering around but feels a bit wooden, at least to me.

Some years ago when I was a student at Harvard Divinity School, I attended a job talk given by the late Ellen Aitken. Ellen discussed the use of the word polytropos in the opening lines of Epistle to the Hebrews, and made a clever case that the anonymous writer of the letter was deliberately fashioning a connection between Odysseus and Jesus for his ancient readers.

Take a look at the letter’s opening verses, and see if you can spot it:

Long ago God spoke to our ancestors in many and various ways by the prophets, but in these last days he has spoken to us by a Son, whom he appointed heir of all things, through whom he also created the worlds. He is the reflection of God’s glory and the exact imprint of God’s very being, and he sustains all things by his powerful word. When he had made purification for sins, he sat down at the right hand of the Majesty on high.

What the NRSV’s translators have rendered as “various ways” is the same Greek word that Wilson renders as “complicated.” The biblical passage goes on to reference the earthly sojourn of the kingly Jesus, and how he suffered, and how he saved his people, and how he eventually returned home to his rightful throne.

In Ellen’s reading of these two passages alongside one another, she has the writer of Hebrews intentionally setting up an intertextual relationship between the wandering and suffering king Odysseus and the wandering and suffering king Jesus. There’s also a bit of an implied contrast, because, as The Odyssey’s intro points out right after our quote cuts off, Odysseus tried and failed to save his men, whereas Jesus succeeded in saving us all.

One can imagine the writer of Hebrews carefully dialing in his word choices — turning the knobs for word selection (polytropos) and word sequence (left a royal home, suffered, saved, returned home) like Rick Rubin at the mixing board, until the figures of Odysseus and the figures of Jesus begin to take focus and then overlap one another in his audience’s minds.

Certainly the writer of Hebrews could have used a word other than polytropos in the opening of his letter, but it wouldn’t have done the same rhetorical work of juxtaposing Odysseus with Jesus. The word selection was an intentional part of the communicative act; it was “load-bearing,” to use a hated Claudism.

The AI watermarking controversy

I’ve gone to a lot of trouble above to make a point that most of us instinctively understand, i.e., that word selection matters a great deal for a text’s full impact on its audience. So when a brother in technology mansplains us that Anthropic’s watermarked text has “no loss of quality” when compared to its non-watermarked text because math, we might be tempted to respond with something like: “No, two differently worded texts are not identical, you absolute giganerd clown. Why don’t you go back to talking someone’s ear off about trains and leave matters of taste to people who have some.”

Arvind Narayanan@random_walker·Aug 18Something amazing is happening in the debate about text watermarking in Claude. This is my best attempt to make sense of the quickly evolving situation.

The key thing to keep in mind is that it is in fact possible to watermark LLM-generated text without degrading output qualityShow more608037281K

I mean, look at the following text from a pretty good explainer on the watermarking issue:

Now the prompt is: today’s weather is “cold,” and a possible answer could be, for example, “gray” or “overcast”. So in contrast to the “Berlin” example, I would say “gray” and “overcast” kind of are interchangeable.

They are both reasonable next tokens for this prompt, given the goal of completing this text or writing the next token. So it’s almost like a coin flip which one we want to select. There is not really an objectively worse one of one or the other.

The kind of person who writes a painfully awkward sentence like, “There is not really an objectively worse one of one or the other”, about the difference between two adjectives is probably the kind of person who has at best a very underdeveloped awareness of craft in writing.

For the rest of us, the issue of word choice is intimately linked to questions of taste, identity, and authorship, which are themselves linked to questions of ownership, status, and profit.

This gut-level intuition is well captured in this response to the watermarking decision:

Jon Teets @JonTeets005·Aug 19Replying to @xlr8harder“The best words in the best order“ is a famous definition of poetry by Samuel Taylor Coleridge.

Eliot argued that the distinction between prose and verse is not clear, and that the best prose often uses poetic structures.

Watermarking corrupts. It rests on the assumptionShow more128399

As for the less vibes-baesd authorship/ownership objection to Anthropic’s watermarking, Ben Thompson ably sums it up:

I am deeply philosophically opposed to watermarking in the context of the entire meta question about the relationship between humans and AI. Implicit in the E.U.’s regulation is the idea that an AI is an independent entity that needs to be distinguished from humans; the alternative view — that I hold — is that AI is (at least for now) a tool that is wielded by humans. From this perspective, to insist on watermarking is no different than insisting that a ballpoint pen advertise itself as the author, a concept that is clearly absurd.

The complaint, then, of the anti-watermarking camp might be paraphrased as follows:

I use an LLM to communicate with other humans; I may not be the kind of person who’s skilled enough at writing to know when to use “gray” vs when to use “overcast”, but I do know that there’s a difference and that my readers will feel the difference; so I’m trusting this tool to pick the best words for my text. But now you’re telling me that instead of always picking the best words, you’re going to sometimes pick the second best word, and you’re going to do this because you want to rob me of my authorship and ownership of this text and claim it for yourself.

Of course, whatever their public insistence to the contrary, I think AI people actually share the nontechnical person’s common-sense view that similar words are not, in fact, freely interchangeable. The fact that some feel need to deny or downplay this intuitively obvious fact points at a tradeoff I think we’ll be seeing a lot more of as LLMs eat the software that’s eating the world.

Before I can explain what the tradeoff is, I should give some background on the two concepts on either side of it.

Verifiable and unverifiable domains

All the AI people pushing back against the watermarking backlash are doing so at least partly on the basis of a mathematically informed understanding that the AI is not trying to pick “the best word” because when it comes to word selection (as opposed to, say, next chess move selection), “best” exists in the domain of the unverifiable.

There is no universally applicable, objective measure of “the best next word” that an LLM training run can target for optimization, because “best” (at least, for our folk-hermeneutic, author-centric purposes here) is entirely contingent on the kind of impact the author intends to have on her audience.

The “best word,” insofar as that concept can even possibly makes sense, is governed by who the author is at a particular moment in time, and who she’s speaking to, and what she’s trying to do with her speech acts.

To develop a better intuition about this problem by translating it into another domain, imagine you’re playing dodgeball in middle school, and you have the ball in your hands. On the opposing team is Susie, a girl you have secret crush on. You are really, really bad at ball-throwing, and you know it.

Now consider these two scenarios:

• You throw the ball, intending to hit Susie, but you accidentally hit Biff full in the face. Biff is now going to beat you up at the bus stop for this insolence.

• You have an Iron Man glove that is deadly accurate at ball-throwing. The glove’s Jarvis bot doesn’t know you like Susie, but it does know that Biff beat you up yesterday. So you throw the ball, intending to hit Susie, but the glove sends the ball directly into Biff’s face… .

Who is the best dodgeball target, Susie or Biff?

The naive answer might be, “Susie,” because that’s who you intended to hit. But if you had hit Susie, maybe she’d dislike you and start avoiding you, and you’d come to realize she was the worst target. But then maybe this would teach you a lesson about girls, and you wouldn’t target your next crush on the dodgeball court, and as a result you’d ultimately succeed in winning the second girl’s affection. Or, if you hit Biff via the glove and he tries to beat you up, well the glove can surely teach Biff a lesson at the bus stop, and possibly make you the school hero… or maybe you get sent to a kind of juvie for supervillains. Who the heck knows?

My point: Even given unlimited dodgeball skill and total knowledge of both intended and unintended consequences, the “best dodgeball target” is only possible to theorize with reference to your own private intentions and circumstances at a given point in time.

You may intend one thing prior to the act, then regret it shortly afterwards. Or, maybe it’s not even possible for you to really say for certain what you wanted when you acted; maybe all you can verifiably know is how happy (or unhappy) you are right now with the results… and even that is subject to change as consequences unfold over time.

The best dodgeball target is deeply, profoundly unverifiable. And no amount of labor on the part of AI benchmark makers is likely to move it into the realm of the verifiable.

The same is true, of course, of “the best word.”

Or wait… is it? Even if there’s no “best word,” some word choices are obviously better than others, and everyone knows this, right?!

…Aaaannnd, now we’re now in the teeth of a very old problem, which we’re not going to solve here. But at least we have enough background to better hone in on the core difference between AI watermark disrespecters and respecters:

• An AI user who hates the watermarks fears that the AI will not pick the best words, whatever those are, for her particular context and intentions.

• An watermark-loving AI worriers fears that the AI might actually pick the best words for the AI’s context and (highly suspect) intentions; so we at least have to know verifiably when the AI is speaking.

Or to rephrase the pro-AI watermark position yet again, this time in the AI field’s own terms:

The issue of ‘best word’ is unverifiable, so why do you even care? But the issue of provenance is mathematically verifiable, and many people who are worried about AI’s plans and intentions care a great deal about whether text is AI-written or not, so let’s at least do the verifiable thing that we care about (i.e. the watermark).

Where does this leave us

I think the pro- and anti-watermark camps are actually driven by the same core set of beliefs about language:

• Word choices really do matter, and similar words are not actually fully fungible (all Anthropic or AIsplainer handwaving about “gray” vs. “overcast”, aside).

• There are better and worse word choices in a given context.

• The ability to pick the words that will have the precise impact you intend, is powerful and potentially dangerous.

• Given the above, the intentions of the party selecting the words matters.

• To the extent that we have to pick between possibly giving some unknown number of users the second-best word (totally unverifiable) and giving society mathematical proof of provenance (100% verifiable), we should obviously pick the latter optimization target.

That last bullet contains the tradeoff I mentioned above:

⚖️ When one value is unverifiable (craft, the “best word”) and a competing value is verifiable (provenance), the technocapital machine will always optimize the verifiable one at the expense of the unverifiable one.

To the extent that there’s no way to quantify how many (if any) users are getting the second-best word, or of measuring the cost to those users of getting a suboptimal word, the confusing and inconvenient concept of “the best words in the best order” may as well not even exist.

(Poor poets. Everyone knows what they mean but nobody knows what they mean. No wonder they’re often poor and suicidal.)

I might as well end by framing this tradeoff as a warning: To the extent that there are true and valuable things in human experience that cannot be measured, we risk losing sight of those things in every place we introduce optimization pressure; and LLMs will expose whole new areas of human experience to optimization pressure.

As with every technological revolution going back at least to writing (Plato had very mixed feelings about the written word), it’ll be up to us to decide what we preserve and what we allow to fade away.