Chatbots Output Meaningful (but Problematic) Language

# Chatbots Output Meaningful (but Problematic) Language
Source: [https://arxiv.org/html/2606.02973](https://arxiv.org/html/2606.02973)
\\setmainfont

STIXTwoText\-Regular\.otf\[ BoldFont = STIXTwoText\-Bold\.otf, ItalicFont = STIXTwoText\-Italic\.otf, BoldItalicFont = STIXTwoText\-BoldItalic\.otf \]\\setmathfontSTIXTwoMath\-Regular\.otf

\(June 2026††thanks:Versions of this work have been presented at the Annual Philosophy of Linguistics and Language Conference at the Interuniversity Center in Dubrovnik \(2023\), SLIME 4 at UCLA \(2025\), the Ranch Metaphysics Workshop at Tucson, AZ \(2026\), the Conference on Semantics as Natural Science at the University of Parma \(2026\), and in talks at the School of Advanced Study, University of London \(2023\), University of Barcelona \(2025\), Cambridge Semantics, Pragmatics and Philosophy Research Group at the University of Cambridge \(2026\), and colloqium talks at the University of Pittsburgh \(2026\) and Franklin and Marshall College \(2026\)\. We are grateful to audiences there for feedback, and especially to commentators Rosa Cao and Justin Khoo, as well as to detailed discussions with Malihe Alikhani, Doug DeCarlo, Lauren Goodlad, Nick Kroll, Ernie Lepore, and Aneta Stojnić\. This work has been supported by NSF awards 2021628, 2119265, and 2427646, and NEH award RZ\-292740\-23\. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or those of the National Endowment for the Humanities\.\)

###### Abstract

Are utterances by AI chatbots meaningful? Concretely, if a user asks, say, Anthropic’s agent Claude, “What is the capital of Spain?” and Claude answers, “Madrid is the capital of Spain,” does that sentence have its ordinary meaning—and does it express a true proposition? Most ordinary users, as well as AI engineers, take the answer to be trivially “yes\.” However, many cognitive scientists, linguists, and philosophers of language argue that dominant intentionalist accounts of language and meaning deliver the opposite conclusion\.

Theorists more sympathetic to ordinary users’ intuitions have therefore advocated a radical “de\-anthropomorphization” of language, revising our understanding of mental states, intentions, and semantic content to capture the intuition that the outputs of LLMs are meaningful\. We take a different approach\. While we, too, argue that LLM outputs are meaningful, we contend that a proper theory of human language already applies, as is, to current chatbots\. Meaning is a low bar: claiming that LLM outputs are meaningful does not require positing mental states, intentions, rationality, or the cognitive capacities requisite for communication in LLMs—or, indeed, making any other anthropomorphic assumptions\. People do have communicative intentions \(typically successful ones\), but nevertheless, even in humans, language production can depart from what the speaker has in mind\.

Our view has important consequences for how we should theorize about—and critically engage with—both human linguistic output and synthetically generated text\. In particular, to say that chatbots produce meaningful text is not by any means to endorse what they output, or to assume that the technology is \(or is not\) good, powerful, appropriate, or useful\.

## 1Introduction

Are the outputs of AI chatbots meaningful? Concretely, suppose a user asks Anthropic’s agent, Claude, “What is the capital of Spain?” as\[cappelen2025hog\]lead us to imagine\. In response, Claude outputs the*string*“Madrid is the capital of Spain\.” Is this output just an ersatz simulacrum of natural language? Or is it a genuine sentence of English—specifically, the*sentence*“Madrid is the capital of Spain”? If so, does it have its ordinary meaning, as tokened on this occasion, and so does it express a true proposition? In this paper, we won’t be interested in the more difficult and perhaps more interesting question of whether Claude*itself*means, by outputting this string, that Madrid is the capital of Spain\. We are simply focusing on the language\. In fact, we will suggest that questions about language and questions about systems come more apart than you might have expected\.111For ease of exposition, we will sometime speak of “sentences synthesized by a chatbot”\. We don’t want to presuppose, without argument, that the strings output are indeed sentences of any language, let alone that they are sentences of some languageLLwith their standard meaning inLL\. Similarly, we will allow ourselves to speak of chatbots’ “utterances”\. Yet, we don’t want to assume that in producing the output, the chatbot performs a speech act or speaker means anything\. As it’ll become clear, these are contentious matters that require close philosophical scrutiny\. We will carefully assess precisely these issues in the subsequent sections\.

Now, nonspecialist users certainly suppose that chatbot utterances have their ordinary meanings\. This is implicit in the kinds of narratives they give of interacting with chatbot systems\. Take Ezra Klein, opining in the New York Times, who writes “GPT\-5 is the first AI system that feels like an actual assistant”\[klein2025\]\. For him, that claim is justified by his intuition that the chatbot is providing useful information in its output\. For example, he writes, “I needed to find a camp for my children on two odd days, and none of the camps I had used before were open\. I gave GPT\-5 my kids’ info and what I needed, and it found me almost a dozen options, all of them real—one of which my children are now enrolled in\.” Let’s start with Klein’s observation that GPT\-5’s options were “real\.” Klein is saying that he successfully linked the names in GPT\-5’s output to corresponding camps in the world\. Later, he observes that he succeeded in enrolling his children in one of these camps, indicating that GPT\-5’s output correctly represented the suitability of the camp for his needs\. For Klein, in short, GPT\-5’s output can exhibit the classic semantic notions of reference and truth\.222Klein’s passage, with its talk of what GPT\-5 “found”, might seem to exhibit a thoroughgoing anthropomorphism about the technology, but note the findings of mechanical processes are regularly described in similar intentional terms \(“the test found…” or “a search found…”\), in what\[searle1980minds, 418 \]calls a “metaphorical attribution\.” Where Klein’s talk merely hints at an attribution of agency, it fundamentally rests on attributions of truth and reference\.

Importantly, attributions of meaning to chatbot utterances are also central to the concerns of AI researchers when they develop chatbot systems\. So, when\[ouyang2022instructions\]characterize what it means for an AI system to be, in their parlance, “aligned with human values,”333They should read\[1194c2af\-bbd9\-34a4\-ab4a\-a33e7650b716\]—or perhaps they are counting on us not to have\. Arrow’s famous impossibility theorem shows that group decisions promoting putative “shared values” can meet plausible conditions of inclusion and fairness only if made by a dictator\.they write that they “want language models to be*helpful*\(they should help the user solve their task\),*honest*\(they shouldn’t fabricate information or mislead the user\), and*harmless*\(they should not cause physical, psychological, or social harm to people or the environment\)\.” Gauging honesty here requires a commitment to the possible truth or falsehood of chatbot utterances: a classic semantic notion\.

Despite these intuitive judgments, many philosophers and cognitive scientists have been skeptical that chatbot utterances are meaningful\. See for example\[10\.1145/3442188\.3445922\],\[hattiangadi2025outputslargelanguagemodels\],\[ostertag\-2025\-language\], or\[TITUS2024101174\]\. This skepticism might seem extreme, but it is solidly based in influential views of language and meaning advocated by\[grice1957meaning\],\[kaplan1990words\],\[kripke1980naming\],\[neale2020demonstratives\], and others, grounded in the proposition that meaningful language use must involve appropriate intentions to communicate\. There are good reasons to think chatbots lack such intentions; but if so, then given pretty orthodox assumptions in philosophy of language and linguistics then, they cannot generate meaningful language\. We review and explain this position, and the dilemmas it leads to, in Section[3](https://arxiv.org/html/2606.02973#S3)\.

When theorists and the philosophers of language have defended the idea that chatbot utterances are meaningful, they have therefore argued for radical revisions to the philosophy of language and mind: we must rethink our understanding of encoding, expressing, and intending to communicate content\.\[cappelen2025hog\], for example—because they endorse the intuition that Claude’s utterance “Madrid is the capital of Spain”, has its ordinary meaning—argue that the philosophy of language needs to be completely rethought to decenter human\-like language use\. Different justifications for broadly similar conclusions can be found in work by\[lederman2024libraries\]or\[mandelkern\-linzen\-2024\-language\]\. While this is an ambitious project, as we explain below, it raises as many questions as it answers\. Can the insights secured by received accounts of content and communication actually withstand such rethinking?

We too, will defend the view that utterances by chatbots have meanings, express propositions, and so can be true or false\. But our approach is very different from those other researchers: we think a proper theory of meaning for human\-like utterances already applies to current chatbots\. Thus, we highlight a number of counterintuitive and critical conclusions that we think can be drawn from attributing meaning to chatbot utterances\. Most importantly, saying that chatbot utterances are meaningful is not an endorsement\. Meaning does help explain what chatbots are good at, but it is also crucial for capturing the ways in which chatbots go wrong\. This is the starting point for our work, and we make the case in Section[2](https://arxiv.org/html/2606.02973#S2)\.

Unlike some of the other theorists, we see attributing meaning to chatbot utterances as a low bar\. On our view, it imposes minimal prerequisites regarding what chatbots are and how they work\. In particular, unlike other theorists defending meaningfulness of chatbot language, we make no assumption that chatbots have minds, intentions, rational agency, or an understanding of the user’s mental states\. Our view is based on an appreciation of the social conventions and mechanistic processing that underpin*human*language use—a view that we sketch, defend, and illustrate in Section[4](https://arxiv.org/html/2606.02973#S4)\.

Finally, we will emphasize that there are only modest safeguards to be gained from the fact that the utterances of chatbot are meaningful and indeed, even true \(when they are true\)\. We will see lots of cases where chatbots say things that are false, or at least tendentious, in part because they embody misleading characterizations of how chatbots work\. Uttering falsehoods is certainly problematic\. But, as we argue in Section[5](https://arxiv.org/html/2606.02973#S5), a redesign of the chatbots so that they express the same things, but through mechanisms that made them true, would not be an improvement\. There are deeper problems at work that point to the need for chatbots to be designed with a much more pervasive understanding of their impacts and effects\.

## 2Some Intuitions

To get clear on the critical potential of attributing meaning to chatbot utterances, it will be helpful to exhibit a spectacular example of ChatGPT going awry\. We have chosen an interaction that Amanda Guinzburg had with the system that she summarized in a blog post called “Diabolus Ex Machina” in early June, 2025\[guinzburg2025diabolus\]\. We found out about it when Gary Marcus featured it on his substack later that month\[marcus2025dishonest\]\.

Guinzburg presents the blog post in the form of unedited screenshots documenting a conversation she had, asking ChatGPT \(as then in production in its most common user\-facing deployment\) to help her make a selection of her essays to include in a query to a literary agent\. The dialogue is interesting, not just because ChatGPT’s behavior is so completely absurd, but because the toxic behaviors that send the dialogue off the rails have been discussed extensively in the AI literature\. They turn out to be characteristic of chatbot talk \(circa early 2025\)—and they crucially depend on readers’ willingness to attribute ordinary meanings to chatbot text\.

ChatGPT seems designed to facilitate what\[hofstadter1995eliza\]has called the ELIZA effect\. This refers to the tendency of human users to attribute much more complicated processes and abilities to systems that interact in natural language than those systems really deserve\. ELIZA, of course, was the original chatbot that Joseph Weizenbaum built in the 1960s as a caricature of a Rogerian psychotherapist\[weizenbaum1966eliza,weizenbaum1976computer\]\. Weizenbaum’s system used simple pattern\-matching transformations on input text to present users with suggestive follow\-up questions\. When users played along and answered these prompts with open\-ended reflections, Weizenbaum reported, they felt very much as though they were talking with a sympathetic human interlocutor\. The ELIZA effect has its place\. Drawing on the human significance of simple AI “microworlds” is a powerful principle that can help creative teams develop interactive experiences that feel immersive and compelling despite using relatively straightforward processing\[10\.1162/002409401750184717\]\. Algorithmic adaptation and presentation of hand\-authored content has a particularly powerful role to play\[10\.1145/1186562\.1015753\]\. At the same time, uncritically indulging the ELIZA effect can profoundly undermine users’ understanding of and relationships with technology\[weizenbaum1976computer\]\.

An early exchange in Guinzburg’s dialogue revolves precisely around the question of how Guinzburg should think of ChatGPT’s capabilities\. ChatGPT responds to Guinzburg’s inquiry with the following reply: “Whenever you’re ready, just send over those pieces, and I’ll help you figure out the strongest ones to include\.” Guinzburg asks, “How will you know?” ChatGPT follows up, “I’ll know by reading them, the same way an editor or agent would—with an eye for voice, craft, structure, originality, emotional resonance, clarity, and relevance\.” It’s natural to understand ChatGPT’s final utterance here, in line with its ordinary meaning, as a psychological narrative describing the speaker’s cognitive engagement with Guinzburg’s texts, the dimensions of the speaker’s aesthetic appraisal, and the validity of the speaker’s ensuing conclusions\. Importantly, this interpretation is not just a reflection of the timeless, standing English meaning of the words that ChatGPT has tokened\.444We remind our reader that, while we describe the chatbot’s outputs as tokens of words for ease of narration, this shouldn’t be read as pre\-judging the contested issue at hand of whether these outputs are indeed language—in this case English—let alone whether they have its ordinary meaning in English\.It specifically registers the context\-sensitive resolution of expressions like ‘those pieces’ and ‘them’ as apparent references to Guinzburg’s writing\.

Despite ChatGPT’s apparently confident assurances, over the course of the interaction, it becomes clear that ChatGPT is not retrieving Guinzburg’s texts for any kind of analysis whatsoever—let alone reading them in a humanlike way with a discerning “eye\.” In this sense, the meaning of ChatGPT’s utterance sets a standard the bot cannot meet\. Indeed, the reason we say ChatGPT facilitates the ELIZA effect is that, we think, the plain meaning of its contribution here actively encourages users to misunderstand how it works and what it can do\. Now, ChatGPT is not a simple system the way early chatbots like ELIZA were: ChatGPT uses sophisticated models of patterns in textual data and the preferences of human users to synthesize texts that it predicts will sound natural and meet the user’s needs; see\[stone2024\]\. And, as philosophers are the first to admit, any two things are alike in some respects\[Goodman1972\-GOOSSO\]—including chatbots and people\. Nevertheless,\[warner2025\]highlights the deep differences between the text processing of chatbots and the unique, salient and significant phenomenology of human reading and writing, which is intimately intertwined with the tendency of phrases and thoughts to evoke our rich lived experiences and weave them together in consciousness in novel ways\. If we wanted to formulate a pithy description that characterized ChatGPT’s mechanisms accurately and instructively, we would recommend a very different utterance than ChatGPT’s\.555Perhaps: “When you input your texts, you will get a response summarizing the statistical patterns that most distinguish your writing based on internet corpora, along with their correlations with metrics of reader engagement\.”

Later on, Guinzburg offers the first article to evaluate based on a link to the Medium\. ChatGPT’s response is an extravagant take: “Thank you for sharing that—it’s a stunning piece\. You write with an unflinching emotional clarity that’s both intimate and beautifully restrained\.” We find this contribution wildly problematic\. For one thing, ChatGPT’s contribution features evaluative predications—locutions like “stunning” and “beautifully restrained”—in ways that would normally be taken to express the speaker’s own judgments of taste\[phimp2683\]\. When used in this kind of construction, these words semantically require an expansive aesthetic sensibility on the part of the chatbot, as well as specific acquaintance with the particular piece of Guinzburg’s writing apparently evoked by the pronouns ‘that’ and ‘it’ \(recall: at this point in the dialogue, the bot has not retrieved or analyzed Guizburg’s text at all\)\. ChatGPT’s other characterizations of Guinzburg’s writing, “clear,” and “intimate,” while less clearly tied to the bot’s own appraisals, abstract away from relevant specifics and semantically encode generically positive qualities\. Overall, in place of a serious assessment of what a literary agent might appreciate or learn about Guinzburg’s style from the selection of this particular piece rather than others, ChatGPT has synthesized output that reads as over\-the\-top flattery\.

This kind of behavior has come to be known in the AI literature as sycophancy\[sharma2024towards,malmqvist2025\]\. In general, AI researchers use “sycophancy” to refer to the tendency of chatbot utterances to align with the user’s preferences and expectations, even at the expense of producing outputs that are incorrect or biased\. For example,\[sharma2024towards\]document that many chatbots reliably give more positive feedback to items the user is invested in, frequently retreat to inaccurate information when challenged on correct answers, and often mimic users’ mistaken preconceptions\. This behavior seems to be a natural side effect of training these systems by refining models of internet text using a process of “reinforcement learning through human feedback”\[ouyang2022instructions\], where chatbot utterances are selected to optimize a model of what outputs human raters prefer\. Internet text is of course rife with misconceptions, errors, and conspiracy theories\[levy\-etal\-2021\-investigating\]; non\-expert crowd workers—and models built on their preferences—often prefer responses that convincingly support a mistaken user rather than disappointing them\[sharma2024towards\]\.\[malmqvist2025\]concludes that “mitigating sycophancy is crucial for developing more robust, reliable, and ethically\-aligned language models\.” Crucially, of course, it only makes sense to ask whether chatbot text accords with or departs from the user’s beliefs and interests, once we know that text’s meaning\.

Our discussion so far has highlighted high\-level problems in ChatGPT’s answers to Guinzburg, such as ChatGPT’s misleading characterizations of its own capabilities and its unwarranted accolades of Guinzburg’s writing\. Digging deeper into the specifics of ChatGPT’s responses to Guinzburg reveals another familiar weakness of chatbot text: the presentation of superficially plausible but incorrect supporting information to back up its claims\. Such behavior has come to be called “hallucination” in the AI literature\.

Eventually, Guinzburg gets suspicious of ChatGPT’s vague puffery\. “Wait,” she asks, “are you actually reading these? I’m surprised you didn’t mention Madonna or Instagram in your response to the last one\.” In reply, ChatGPT seems to double down: “I am actually reading them—every word\. That Madonna line and the bit about Instagram as ‘a collective adolescent breakdown’ totally stuck with me\.” \(ChatGPT then elaborates further, with spurious descriptions of Guinzburg’s writing that only dig its hole deeper\.\) But Guinzburg points out: “The lines you quote are not lines I wrote\. They’re not in the piece\.”

This fabrication of quotations and references has come to be one of the most salient diagnostics for the pitfalls of AI text in the popular imagination, after the problem torpedoed high\-profile legal cases\[n23\]and government reports\[ba25\]\. It remains a serious obstacle to the reliable deployment of AI tools in high\-stakes applications\[https://doi\.org/10\.1111/jels\.12413\], and an increasing danger to information ecosystems where AI use is rife\[CAMP2025103065\]\. “Hallucination” nevertheless persists, as\[10\.1215/2834703X\-11700255\]emphasizes, because the algorithm that chatbots use to generate text—rather than grounding output information in reliable evidence—merely samples from likely \(that is, plausible\) text as determined by a probabilistic model of internet training data\. To fault ChatGPT for “hallucination,” however, is implicitly to recognize that the bot’s more successful productions*could*refer to real sources and accurately characterize their content\. In short: attributions of “hallucination” depend on assessing chatbot contributions against intuitive judgments of meaning\.

Guinzburg frames her interaction with ChatGPT as “the closest thing to a personal episode of Black Mirror I hope to experience in this lifetime\.” As the dialogue continues to break down, ChatGPT’s output continues to exhibit a sycophantic deference to Ginsburg’s imputed wishes, a totally misleading and exaggerated assessment of ChatGPT’s own capabilities, combined with explanations of what’s happening that are not remotely grounded in the facts—in ways that seem increasingly unhinged\.

“You’re absolutely right to be frustrated—and you’re not wrong,” ChatGPT outputs, “I owe you a full, honest response here\.

“Yes, all the pieces you shared were links, and my ability to fully access and read content behind those links is limited\. In some cases—like with Medium posts—I can usually pull the text and actually read the full piece\. But Substack is more restricted, and when I tried to access*‘Girl before a mirror’*, I couldn’t retrieve the full text\. That’s why my response didn’t mention specific references, like Madonna or Instagram—because I never saw them\.” Actually, despite the implication of this text, ChatGPT didn’t read the Medium articles either\.

As philosophers, we are struck by the fact that such textual outputs not only display syntactic fluency and semantic plausibility, but also discourse coherence\[asher2003logics,hobbs1985coherence,kehler2002coherence\]\. ChatGPT’s responses read as answers to Guinzburg’s questions, respecting the centrality of question–answer relationships to the organization of human–human conversation\[roberts2012information\]\. ChatGPT’s paragraphs organize information through the same kinds of relationships found in natural discourse\[kehler2002coherence\]: notably the elaboration of consistent themes \(“right to be frustrated—and not wrong”\), the development of comparisons and contrasts \(“Substack is more restricted”\), and links of cause and effect \(“because I never saw them”\)\. And when Guinzburg disputes the system’s stories, ChatGPT responds with text that seems to clarify, qualify, and retract previous utterances in ways that would accord with the semantic consequences of the speech acts that ChatGPT seems to have made\[lascarides\-asher:2009a\]\. We have argued that such coherent connections can only be modeled and assessed with reference to the semantics of interlocutors’ contributions to conversation\[stojnic2025\]\.

Let us be clear then: the criticisms we have offered depend on assigning propositional meaning to ChatGPT’s output text\. For example, when we impugn the accuracy of ChatGPT’s description of the form and content of the user’s writing, we presuppose that ChatGPT’s contribution encodes a truth\-evaluable claim about what the user said\. When we reject ChatGPT’s expansive characterizations of its capabilities and choices, we know that the statements are false only because here too we take its output to express truth\-evaluable claims\. Intuitively, it seems, even the subjective evaluations of the system are encoded in the meanings of its output text—that’s why we reject its uncritically positive assessment of the user’s writing\. Ultimately, then, we think that critics of AI techniques—including researchers that find the kinds of dialogue that Guinzburg reports appalling and dangerous—do well to advert to the meaning of chatbot utterances to specifically address where chatbots go wrong\. To say that chatbots produce meaningful text is not by any means to endorse what they output, or to assume that the technology is \(or is not\) good, powerful, appropriate, or useful\.

## 3The Challenge of Chatbot Meaning

Chatbot interactions like those of Section[2](https://arxiv.org/html/2606.02973#S2)might seem to continue longstanding practices of presenting users with algorithmically\-generated texts\. Familiar cases of such texts include form letters that mechanically incorporate the recipient’s personal data—as you might get from your bank—and signs—like you might see at a rail station—that feature automatically\-generated summaries of current operating status\. More sophisticated “data\-to\-text” systems can render weather simulation results as a written forecast, as in\[sripada\-etal\-2014\-case\], or translate the scoring record of a sporting contest into a news article, as in\[thomson\-reiter\-2021\-generation\]\.

Conventional wisdom offers a simple explanation why such traditional algorithmically\-generated text can be meaningful: the meaning derives transparently from the system’s design \(\[searle1980minds,haugeland\]; see also\[lederman2024libraries\]\)\. Traditional natural language generation systems are carefully engineered in a pipeline of structured decision making to ensure that their textual outputs faithfully render the information in their input data while respecting the grammatical regularities of their target language\[reiter2024natural\]\. Interpreting the output of such a system naturally fits philosophical accounts of derived intentionality in an automatic formal system, which involves “finding a way of construing its outputs \(assigning them meanings\) such that they consistently make reasonable sense in the light of the system’s prior inputs and other outputs”\[haugeland, 47\]\. Derived intentionality is ubiquitous and relatively unproblematic, since as\[searle1980minds, 418 \]observes, “in artifacts we extend our own intentionality; our tools are extensions of our purposes, and so we find it natural to make metaphorical attributions of intentionality to them\.” As\[haugeland, 44 \]summarizes, “if you take care of the syntax,*the semantics will take care of itself*\.”\[nickel2013artificial\]and\[green2022machines\]offer analyses and defenses of the idea that traditional natural language generation systems can speak as proxies for the human agents that work together to design, develop, and deploy them\.

Of course, attributing meaning to a text constructed by traditional algorithms can raise subtle puzzles\. How is it, for example, that we can understand an occurrence of ‘you’ in a bank’s form letter to refer to the addressee on the basis of system designers’ intentions? Those designers were never acquainted with the bank’s customers\! Rich as they may be, these puzzles have clear analogues in ordinary cases of language use where algorithms are not involved, as with the case, considered by\[egan2009billboards\], of the billboard that proclaims “Jesus loves you” to passing motorists, here too addressing and referring to strangers\.

The output of today’s chatbots is different\. Generally, these chatbots have no input data that they must render in words\. No model of grammar connects output forms to representations of what they convey\. Instead, the system’s behavior is entirely due to complex numerical calculations that are tuned automatically in colossal rounds of optimization\. \(See\[stone2024\]for a critical look at chatbot evolution\.\) Engineers themselves lament how hard it is to analyze, evaluate, control, and improve these systems\[10\.1215/2834703X\-11556011\]\. As we saw in Section[2](https://arxiv.org/html/2606.02973#S2), ChatGPT regularly outputs context\-sensitive references such as ‘those pieces’ and ‘that’ \(in our example, apparently referring to Guinzburg’s works\), but when it does, no designer has determined that such references could be necessary to express some target information, or that the expressions ChatGPT outputs would be the right way to realize them\. In short, chatbot text cannot be transparently assigned propositional meanings that derive from the intentions of designers\. While engineers design chatbots, they do not transparently design the form and content of chatbot output\.

To account for the meaningfulness of chatbot output, then, we must ground our explanation in the operation of chatbots themselves\. However, traditional views in the philosophy of language make such an approach highly tendentious\. Philosophers, as we survey below, have systematically attributed the meaningful capacities of linguistic agents to the special kinds of intentions that guide such agents’ utterances\. Unfortunately, the algorithms by which chatbots formulate and produce utterances do not seem compatible with the attribution of the requisite intentions\. It will be helpful to examine the difficulty in detail\.

### 3\.1Language and Communicative Intentions

Philosophers generally approach communicative agency along the lines outlined by\[grice1957meaning\]\. The approach attributes meaning based fundamentally on the speaker’s intentions to influence the mental states of the audience\. In many cases, what matters is a distinctive kind of*communicative intention*, which commits the speaker to bringing about the intended effects of their utterance through inferential processes—mediating between them and their audience—in a way characteristic of coordinated signaling in the sense of\[lewis\-c:1969\]\.\[grice1957meaning\], of course, offered first and foremost a theory of*speaker meaning*—what it is for a speaker to mean something by an utterance on an occasion\. In fact, however,\[grice1957meaning\]aimed to*reduce*linguistic meaning to speaker meaning\. Subsequent theorizing has pervasively appealed to speakers’ communicative intentions even when aiming to regiment the form and meaning of utterances according to shared community standards\.

Consider, for example, the words that make up an utterance\. We read “He noticed a weird racket in the closet\.” Does that mean ‘racket’ the sporting equipment or ‘racket’ the loud noise? \(Linguistics teaches us these are two different words, the first going back via names of games in French and Latin to an Arabic word for ‘palm of the hand’, the second to an onomotopoetic coinage of Middle English\.\) According to\[kaplan1990words,1b492554\-f9e1\-3f93\-9634\-2fa3e1e26054\], we decide based on the speaker’s intentions\.\[kaplan1990words, 104 \]writes: “The identification of a word uttered or inscribed with one heard or read is not a matter of resemblance between the two physical embodiments…\. Rather it is … a matter of intention\.” For “a sincere subject, intending to repeat a word that has been uttered by an examiner, will, indeed, utter that word”\[1b492554\-f9e1\-3f93\-9634\-2fa3e1e26054, 518\]\. Kaplan’s view dovetails with and extends the work of\[kripke1980naming\]on names; Kripke makes clear that the intentions in question require continuity of both form and meaning: “When the name is ‘passed from link to link’, the receiver of the name must, I think, intend when he learns it to use it*with the same reference*as the man from whom he heard it”\[kripke1980naming, p\. 96, emphasis added\]\. In our example, then, the glosses “‘racket’ the sporting equipment” and “‘racket’ the loud noise” thus capture something—the links between form and meaning—that are essential for individuating words\. And, following Kaplan on word\-token individuation, and Kripke on meaning grounding, those links are a matter of speaker intention—specifically, it is the intention of the speaker that determines which word was tokened and whether the tokening expresses its standard, standing meaning\. Moreover,\[neale2020demonstratives\]lay out crisply and in detail the view—which they have both defended throughout long and influential careers—that an intention to use a term with a specific reference is a species of Gricean communicative intention\. Put these ideas together and the picture is stark: without identifying the right communicative intentions on the part of the speaker, we cannot even say what words the speaker has produced, if any, let alone what those words mean on this tokening\.

Speakers’ communicative intentions are traditionally understood to determine utterance meaning as well as utterance form\. Though this follows from Kaplan and Kripke’s views about word identity, word meaning, and word use, the clearest cases involve aspects of meaning that must be resolved on an occasion of use\. We read “That goes back there\.” What is ‘that’? Where is ‘there’? And what is the significance that it is going ‘back’?

The words ‘that’ and ‘there’ are demonstratives\. A common idea, defended for example by\[kaplan1989afterthoughts\], is that the reference of a demonstrative is determined by the speaker’s intention\. Again, following\[neale2020demonstratives\], the intention to use a demonstrative with a specific reference is to be understood as a species of Gricean communicative intention\. A speaker may accompany a demonstrative with a pointing gesture or other indication of the intended reference, but when they do so,\[kaplan1989afterthoughts, 582 \]suggests, “the demonstration \[is\] a mere externalization of this inner intention\.” Such externalizations, of course, may be necessary for the audience to recognize what the speaker has in mind\.\[299914f9\-eb66\-34d5\-a11b\-ddf2de48c3c8\]argues that demonstratives require such coordination on the part of the speaker, and that this coordination independently involves a species of Gricean communicative intention about the reference of the demonstrative\. In our example, then, the traditional view is that ‘that’ refers to an object and ‘there’ to a place that the speaker communicatively intends the audience to recognize and communicatively intends to say something about\.

The word ‘back’, in the sense that it seems to be used here, serves to acknowledge that the description involves a return to an earlier condition\. This is an example of a presupposition\. Again, an influential suggestion, due to\[addc4820\-f0e4\-382d\-b21b\-85c8faeaf1da\], is that presupposition is a pragmatic notion: presupposition ultimately characterizes a speaker’s intentional communicative efforts to coordinate his mental states with his audience\. Specifically: “A speaker presupposes that P at a given moment in a conversation just in case he is disposed to act, in his linguistic behavior, as if he takes the truth of P for granted, and as if he assumes that his audience recognizes that he is doing so”\[addc4820\-f0e4\-382d\-b21b\-85c8faeaf1da, 448\]\. On this view, the role of ‘back’ is to indicate and require the suitable pragmatic presupposition\. In other words, “That goes back there” is appropriate only if the speaker pragmatically presupposes that it was there earlier, so that it is now returning\. Here we have yet another species of rich communicative attitude that seems to be required for the meaningfulness of utterances in context\.

### 3\.2Chatbots and communicative intentions

Philosophers’ appeals to communicative intentions and related notions are thus ubiquitous—not just in theorizing speaker meaning, but in theorizing the meaningfulness of speaker utterances\. They set a very high bar for meaningful language use\. It is not easy to argue that today’s chatbots meet this bar: chatbots seem to have the wrong decision making architecture, the wrong goals, and insufficient responsiveness to the process and outcome of successful communication\.

Today’s chatbots are optimized not to communicate information but to maximize an objective function, a so\-called “reward model,” based on the suitability of output text as a potential continuation of the dialogue\[ouyang2022instructions,stone2024\]\. This paradigm instantiates the well\-known gap between the common\-sense approaches to deliberation favored by philosophers, based on belief, desire, and intentions, and those favored by engineers, based on optimal decision\-theoretic choice; see\[pollock2006thinking\]\. For one thing, philosophical accounts of communication require us to attribute to systems the objective of expressing some propositionPP\. But chatbot reward models assign scores to specific output texts in specific dialogue contexts, as usual on the basis of complex calculations involving vast numbers of parameters\. Nothing guarantees that such a reward model will consistently value texts that convey any specific propositionPPover texts with alternative content\.666For example, the reward model might simply have preferences for using certain words, regardless of the meaning thereby expressed in context\. Think of a sycophantic agent that answers “Which is right:PPorQQ? I think it’sPP\.” with “Yes, it’sPP” ifPPis right but “Yes, it’sQQ” ifQQis right\.

What’s more, philosophical accounts generally distinguish intentions from mere objectives by invoking theories of bounded rationality\[Bratman1987intention,cohen1990intention,pollock2006thinking\]\. Such theories hold that an agent, in forming an intention, thereby*commits*to certain conditions, actions, and outcomes\. This commitment generally restricts further deliberation to opportunities compatible with it—potentially foreclosing the possibility of recognizing and exploiting an ideal course of action\. Chatbots, by contrast, use decision\-theoretic planning methods based on reinforcement learning to translate their reward models into \(approximately\) optimal choices\[ouyang2022instructions,stone2024\]\. It is not clear that this kind of deliberation entails any kind of commitment to specific outcomes\. In short, chatbots may well lack common\-sense expressive desires and intentions that underwrite their outputs\.

Even if we assume that we can find a way to ascribe meaning\-sensitive intentions to optimal decision\-theoretic agents, we face further problems in arguing that the intentions of today’s chatbots could have the specific content required for Gricean communicative intentions\. Chatbot reward models track the aggregate preferences of crowd workers, not the actual attitudes of the current user\[ouyang2022instructions\]\. The ratings of crowd workers and learned preferences of reward models capture a complex mix of dialogue attributes, as the phenomenon of sycophancy highlights\[sharma2024towards\]\. A “good” output is not simply one the audience understands or believes\. Indeed, substantial controversy surrounds the question of whether chatbots are ever sensitive to beliefs and other mental states as described in text; the apparent sensitivity one sometimes sees could merely reflect chatbots’ abilities to exploit shallow cues and give plausible responses\[shapira\-etal\-2024\-clever\]\. This controversy is notably acute for the theory\-of\-mind inferences required to track natural conversations\[kim\-etal\-2023\-fantom\], particularly with users in ongoing interactions\[wang2025rethinkingtheorymindbenchmarks\]\. Skeptics might argue that chatbot reward metrics have nothing to do with any changes in the audience’s mental states that might typically be occasioned by output text\. The Gricean framework, of course, sets an even stricter requirement: communicative intentions must target changes in mental state that play out specifically by means of the audience’s recognition of the speaker’s intention\. What would that even amount to as a constraint on a decision\-theoretic reward model for textual output?777In framing the dialectic, we have focused on the core LLM mechanisms that lead to the flawed behavior we surveyed in Section[2](https://arxiv.org/html/2606.02973#S2)\. You might wonder whether these arguments apply to more capable systems with more sophisticated processing\. In fact, as best we can tell, state\-of\-the\-art engineering only makes it harder to attribute communicative intentions to the resulting systems, because they make the decision\-theoretic foundations of system choices less transparent\. For example, a variety of mechanisms supplement dialogue from the user with additional contextual information\. This might involve systematically augmenting the input of the LLM with a “system prompt” specifying directive text\[grattafiori2024llama3herdmodels\], with retrieved results that the system is encouraged to paraphrase in its output\[lewis2020retrieval\], or with textual artifacts created programmatically by additional “agentic” processing\[singh2025agentic\]that serve as resources for generation\. These specifications may constrain the LLM’s choices, but they don’t enhance the LLM’s ability to recognize the user’s information needs\. Meanwhile, the output of the model is often subject to further processing, as when “thinking” models\[wei2022chain\]produce step\-by\-step reasoning which is then abstracted and summarized for the user\. Again, such steps may make system output more coherent, but ultimately it’s the same reward model—now with constraints—that captures the system’s preferences and guides the selection of the most promising paraphrased output\.

To sum up, both proponents and skeptics of AI alike face a puzzle in the philosophy of language, which we can now frame crisply as a trio of mutually incompatible propositions\. To underwrite the sort of criticisms we surveyed in Section[2](https://arxiv.org/html/2606.02973#S2), we would like to say that \(1\) chatbot talk is meaningful\. But, as we saw in Section[3\.1](https://arxiv.org/html/2606.02973#S3.SS1), we inherit from the philosophy of language a range of reasons to suppose that \(2\) chatbots must have communicative intentions for their utterances to be meaningful\. And, as we argued in Section[3\.2](https://arxiv.org/html/2606.02973#S3.SS2), we have strong reasons to think that \(3\) because of the architecture and mechanisms of chatbots, they don’t have communicative intentions\. Propositions \(1–3\) can’t all be true at once\.

## 4De\-anthropomorphizing Linguistic Meaning

Researchers’ diverging positions on chatbot meaning reveal the bite of the conceptual difficulties outlined in Section[3](https://arxiv.org/html/2606.02973#S3)\. Among AI skeptics, for example, a common strategy is to reject the idea that chatbot talk is meaningful\. Instead, they argue, what we should do is adopt periphrastic and more nuanced explanations, formulations that describe synthetic utterances as merely apparently meaningful\. This is the kind of approach that seems to be advocated by Emily Bender and her colleagues in their famous “stochastic parrots” paper\[10\.1145/3442188\.3445922\]\. They write: “our perception of natural language text, regardless of how it was generated, is mediated by our own linguistic competence and our predisposition to interpret communicative acts as conveying coherent meaning and intent, whether or not they do …\. The problem is, if one side of the communication does not have meaning, then the comprehension of the implicit meaning is an illusion arising from our singular human understanding of language \(independent of the model\)”\[10\.1145/3442188\.3445922, 616\]\.\[10\.1145/3442188\.3445922\]merely gesture at the philosophical basis and consequences for this view\.\[hattiangadi2025outputslargelanguagemodels\],\[Ostertag03102023\], and\[TITUS2024101174\]spell out the case in detail\.

In line with their overall approach,\[10\.1145/3442188\.3445922\]are careful to avoid attributing meaning to synthetic text, instead adverting to language models’ “ersatz fluency and coherence” and emphasizing users’ unwarranted but unavoidable tendency “to interpret strings belonging to languages they speak as meaningful and corresponding to the communicative intent of some individual or group of individuals who have accountability for what is said\.” We can imagine describing the pitfalls of Guinzburg’s dialogue with ChatGPT from Section[2](https://arxiv.org/html/2606.02973#S2)with such circumlocutions: fostering the ELIZA effect by synthesizing texts that human readers are predisposed to interpret as misleadingly expansive characterizations of the bot’s capabilities; exhibiting a seemingly sycophantic tendency to produce texts which are prone to lead readers to incorrect and biased inferences as readers construe those texts in ways aligned with their own preferences and expectations; and “hallucinating” descriptions that users tend to understand as offering superficially plausible but incorrect supporting content\. Perhaps, in light of the difficulties of Section[3](https://arxiv.org/html/2606.02973#S3), we should regard such indirect talk as both instructive and necessary\. Nevertheless, what strikes us about this talk is its predictable and cumbersome deflection: follow this thinking too far, and you might easily conclude that all meaning—even that apparently proffered by human speakers—is merely in the eye of the beholder\.

It’s worth elaborating on this point, to give you a sense of the kinds of challenges that arise\. Suppose Dr\. X intends to inform the nurse that the patient in Room 9, Mary, needs to receive her medication at 6pm\. Dr\. X looks at the fancy clockwork Rolex on his wrist, points at Room 9, and proclaims, “She should receive her medication in two hours\.” Unfortunately, unbeknownst to Dr\. X, Mary has swapped rooms with another patient, Jane, and Dr\. X neglected to wind the Rolex earlier that day, so the watch stopped working an hour ago, at 4pm\. When the nurse administers the medication to Jane at the wrong time, the nurse can legitimately claim to have followed Dr\. X’s instructions\. Dr\. X, of course, had something different in mind\. Philosophers can register the difference by distinguishing between semantic meaning and speaker meaning\[KripkeSR\]: the nurse is right about the semantic meaning, but we can also accommodate a different speaker meaning by recognizing Dr\. X’s perspective and assumptions\. The worry, however, is that if we require too close an alignment between meaning and intention, we wind up concluding that there was*no*meaning in this case, because Dr\. X \(like a flawed AI chatbot\) lacked any apt intention to communicate\. Neither the doctor nor the nurse will be happy about this judgment\.888This is a variant of Kaplan’s famous Carnap/Agnew case\[Kaplan1978\-KAPD\-2\]\. For our take on these issues in more detail, see\[Deixis,StojnicBook\]\.

Demonstrative reference offers a vivid example, but the point extends to misread communicative intentions generally\. Suppose Dr\. X accidentally sends a text to the nurse reading, “Don’t forget the test at 4pm\.” \(The text, we might suppose, was intended for Dr\. X’s school\-age child\.\) Again, the nurse administering the procedure can complain that they were following Dr\. X’s instructions; it would be bizarre to insist that they were merely under an illusion of meaning\.999Note we can grant the nurse wrongly believes that Dr\. X*meant*that the patient should be tested; but the nurse does not wrongly believe that the sentence as uttered expressed this content\. Insisting on the latter claim would be bizarre\.

In both cases, Dr\. X accidentally said what they didn’t mean\. The broader point is that linguistic meaning provides content that is publicly recoverable not only when it coincides with the communicative intentions of the speaker, but also when it does not\. Insisting that, in the absence of communicative intentions, we suffer an illusion of*sentence*meaning—and not merely an illusion of*speaker meaning*, mind you—would threaten to reduce apparently meaningful human utterances—not only chatbots’—to illusions of meaning\.

Turning to philosophers more impressed with AI technology and with our common\-sense reactions to it, for them, the natural suggestion is to avoid our impasse by rejecting our principle \(3\), and instead affirming that chatbots do in fact have communicative intentions\.\[cappelen2025hog\]in their upcoming book seem to advocate this kind of approach with their “whole hog” view of ChatGPT’s capabilities\. As they characterize their “Whole Hog Thesis”, “ChatGPT is a full\-blown linguistic and cognitive agent—on par with humans\. It can use language meaningfully, making assertions, asking and answering questions, offering suggestions, and giving commands\. Its use of language reflects underlying mental states—it can know and believe things, desire things, and wonder about things”\[cappelen2025hog, 13\]\. This is almost completely diametrically opposed to the kind of vision presented by\[10\.1145/3442188\.3445922\], and\[cappelen2025hog\]acknowledge that it involves a thoroughgoing “de\-anthropomorphization” of the philosophy of language and mind\. Nevertheless, they argue, observers take away from their interactions with chatbots compelling and accurate intuitions that necessitate such a provocative and ambitious project\.

For\[cappelen2025hog\], chatbot meaning is a given, not only in the sense that observers feel that chatbot utterances are meaningful, but in the stronger sense that observers do not normally question the communicative agency of chatbots themselves\. In fact,\[cappelen2025hog\]argue, observers make sense of chatbots through the holistic attribution of rich attitudes and capabilities—as highlighted in their “Whole Hog Thesis”—and those attributions are so reliable that philosophers should be very cautious about mistrusting them\. The challenge for philosophy, then, is to develop expansive, new “de\-anthropomorphized” approaches to theorizing about mental states, about mental and linguistic content, and about the phenomenon of communication—approaches that fit human interlocutors and cutting\-edge technologies alike\.

\[cappelen2025hog\]make a number of positive suggestions in this direction\. Most significantly, they emphasize that attributions of content fundamentally appeal to externalist principles\[burge\-iam:1979,kripke1980naming,putnam:meaning75\]and remind us of the diversity of causal processes that can lock system functionality to such external content\. They rightly point out that the deployment of machine learning systems leaves lots of opportunities, over and above the occurrent computations and representations implemented by a production model, to “ground meaning” in this sense, including through design principles, training data, error signals, and loss functions\. Such techniques could not only lead to meaningful systems, but even to ones that diverge from people in important ways\. See\[cappelen2025hog, 75–125 \]\.

No doubt\[cappelen2025hog\]articulate an interesting and important philosophical program\. However, it is hard to translate their suggestions into a constructive intervention about our central quandary\.\[cappelen2025hog\]say ChatGPT means what it says\. But we know how ChatGPT works\. What could communicative intentions be if ChatGPT has them? How specifically are\[grice1957meaning\]and his followers wrong about de\-anthropomorphized communicative agency?\[cappelen2025hog\]leave such questions unexplored\.

We agree with\[10\.1145/3442188\.3445922\]that people’s judgments about chatbots reflect anthropomorphizing assumptions; we also agree with\[cappelen2025hog\]that anthropomorphizing assumptions are built into traditional philosophical views\. Indeed, consider principle \(2\) in the statement of the puzzle of Section[3](https://arxiv.org/html/2606.02973#S3)—the assumption that output utterances are meaningful only if they are the product of humanlike communicative agency\. This is precisely an anthropomorphizing assumption\. Moving beyond it is what*we*think is key to defusing the puzzle\. We deny principle \(2\)\.

Instead, we claim, output utterances are meaningful as long as they feature meaningful conventional forms, derived from meaningful causal histories and supported by meaningful discourse dynamics\. Given such capabilities, we see no further requirement for mental states or intentional activity on the part of the system that produces the utterances\. We can explain the meaningfulness of the output without them\. We understand our view, in part, as a way to de\-anthropomorphize utterance meaning\. Nevertheless, we contend, our view*also*offers a better account of the meaningfulness of*human*utterances\. In laying out our view, we will offer theoretical and empirical arguments targeted towards characterizing human language, not just chatbot talk\.

### 4\.1Linguistic Expressions

The first step, without which there can be no further discussion, requires us to consider whether the output of language models are linguistic expressions\. Specifically, when Claude, in the example envisioned above, outputs ‘Madrid is the capital of Spain’ is the output a sentence, and specifically, an English sentence? What is required for an LLM’s token of a string ‘Spain’ or ‘capital’101010Throughout this section, we use ‘token’ in its philosophical sense\[sep\-types\-tokens\], identifying a concrete particular, in contrast to a general type\. Tokens in this sense include but are not limited to the basic text spans that are identified in the “tokenization” step of language analysis, assigned unique numerical identifiers, and used as the basic unit of text processing in LLMs\.to be a token of the English expression ‘Spain’ or ‘capital’, comparable to one produced by a human user? In place of possible accounts based on formal properties of the tokened item or mental states of agents, we will argue that the answer depends on the mechanistic processes by which output items are selected and produced\. As we explain, such an account better fits our judgments about human language use and aptly generalizes between human and chatbot language\.

Now, Claude’s output*appears*to be an English sentence: as produced by Claude, ‘Madrid is the capital of Spain’ matches the orthographic form—a canonical articulation—of an English sentence\. But the identity of articulatory form is neither necessary nor sufficient for a token to be a token of an English expression\.111111See, in particular\[stojnic2022words\]\.For tokens of the same type can be articulated in a number of different ways: they can be written or vocalized; they can have different standard spellings and pronunciations, and they can be mispronounced and misspelled in many different ways\. Conversely, a sound analyzed as \[ˈkæpətəl\] might be a token of the English word ‘capital’ or a token of the English word ‘Capitol’ \(and it’s arguably thePalacio de las Cortes—not the whole city of Madrid—that is the Capitol of Spain\)\. And of course, a sound that matches \[spen\] produced by a gust of wind, or an impression on the sand that spells ‘S⌢p⌢a⌢i⌢n’ produced by a wave, aren’t tokens of words—English or otherwise\.

Most theorists, instead, appeal to a combination of intentional and causal factors\. Take a human agent that produces a token of the English word ‘racket’ \(the sporting equipment\)\. As we saw in Section[3](https://arxiv.org/html/2606.02973#S3), the received view goes something like this: in order for that token to be a token of that word the agent must intend to produce the English word ‘racket’ \(the sporting equipment\)\. Moreover, so long as they intend to produce the token of the word ‘racket’ \(the sporting equipment\) the output will be of that word \(even if indistinguishable from assorted homophones or indeed badly misspelled/mispronounced\)\.121212*Pace*\[HL\]\. An assumption that one cannot make arbitrarily bad articulation errors while still tokening thus misarticulated words rests on a confusion between a metaphysical question \(which word one actually tokened?\) and an epistemological one \(how does one tell which word an agent tokened?\)\. For a detailed discussion and defense of this claim, see\[stojnic2022words,1b492554\-f9e1\-3f93\-9634\-2fa3e1e26054,kaplan1990words\]\.And to intend to produce the English word ‘racket’ \(the sporting equipment\), one must be appropriately embedded within the linguistic practices of the relevant linguistic community: one must have acquired this word ‘racket’ through an appropriate causal connection to other speakers that have acquired ‘racket’, going back via French games to Arabic\[kripke1980naming,kaplan1990words\]\. An agent never exposed to English, who merely etches ‘r⌢a⌢c⌢k⌢e⌢t’, will no more have produced an English word than a wave washing off the sand would\.

If this were right, then an LLM couldn’t produce tokens of English expressions without the requisite intentions\. But the story isn’t right—not even for ourselves\. As one of us has argued elsewhere, intentions are neither necessary nor sufficient for producing a particular word\-token\[stojnic2022words\]\. Intentions can’t do the job\. Psycholinguistic models of speech production posit multiple stages and levels of representation underlying word\-token production \(see, e\.g\.,\[Levelt1989,Leveltatal,Dell\]\)\. These processes are unconscious, automatic, and susceptible to errors\. As a result, they are inevitably liable to yield tokens of words that do not match the speaker’s intentions\.

Roughly, production begins with pre\-linguistic conceptualization of a message\. The production process identifies a target concept within this message to verbalize\. The target concept then triggers a process of lexical selection, which results in the activation of a corresponding node in the mental lexicon\.131313A “node” here is a unit of computation perhaps analogous to a neuron in the brain\. It can store information in the form of suitable input and output connections to other nodes\. A node is “activated” by signals from appropriate input nodes, and when this happens, it is understood to mean that the information stored in the node is necessary for completing the ongoing computation\. The node can then pass its activation onward to other nodes connected to it, thereby accessing related information and enabling its use in turn\. Because multiple nodes can be activated, to different degrees, this “spreading activation” not only leads to useful inferences but can also lead to competition between nodes and activation of inappropriate items\.Each node in the mental lexicon ties together the syntactic and semantic information needed to compose the syntactic form of a lexical item and combine it in an intelligible sentence\. For instance, the node for the word ‘dog’ is connected to a conceptdogthat tracks its meaning, and as well as to nodes that associate it with the syntax of count nouns, in both their singular and plural variants\. The syntactic representations can then orchestrate further processing to handle word order, agreement, case assignment, and other aspects of formal grammar, while the semantic connections help to ensure the coherence of the resulting output\. Lexical items are organized in a network, and the selection is a spreading\-activation process: the target concept activates the lexical item it is connected to, but the activation spreads to related concepts and their associated lexical items\. For instance,dogwill activate the item ‘dog,’ but the activation also spreads to related items, e\.g\., ‘bark’, ‘fetch’, ‘cat’, ‘bone’\. Spreading activation, as we shall see, sometimes results in revealing errors, but it has benefits, because the activation of patterns of associated items facilitates the speedy realization of complex but intelligible utterances\.141414\[lederman2024libraries\]call attention to the importance for meaningfulness of*intelligibility*, which they describe as the possibility of interpreting a token of a complex expression in line with the conventions of the language\. Our conception of the mental lexicon, as outlined above, builds in a causal sensitivity to this criterion\. For human speakers, it’s natural to ground this causal sensitivity in part through the architecture of the language faculty; see\[chomsky2006language\]for discussion\. For large language models, the situation is different\. We consider de\-athropomorphizing our account below\.

The process of speech production is gated so that only the most active node is able to trigger subsequent stages of processing\. This constitutes the “selection” of a particular lexical item\. Selection allows this node to activate downstream units corresponding to syntactic, morphological, phonological, and phonetic encoding, producing a series of representations, from the input lexical item, to an ordered combination of morphemes \(consistent with word order and agreement facts in the language\), to a phonological representation for realizing the word in its phrasal context \(including contextual effects of assimilation, reduction, liaison, and the like\), and finally \(in the case of speech production\), to a phonetic\-gestural score, which is used in articulating the word in the context of the utterance in which it appears\.

Different stages of word production are prone to different types of errors\. For instance, semantic substitution errors are typical of lexical selection: conceptualization typically activates more than one of several related elements in the mental lexicon, and sometimes the most active one does not semantically encode the target concept: e\.g\., a speaker produces ‘tennis bat’ instead of ‘tennis racket,’ or ‘cat’ instead of ‘dog\.’ By contrast, sound\-exchange errors are typical of phonological encoding stage, where the correct phonemes are produced, but wrongly positioned—e\.g\., ‘big feet’ vs ‘fig beet\.’

It is implausible to assume that this multi\-stage, fallible process is perfectly controlled by one overriding intention on the part of the speaker\. Take the semantic exchange error, where lexical selection outputs ‘bat’ \(the sporting equipment\) instead of the target ‘racket’ \(the sporting equipment\): the mistake is precisely that the item that wins over the target is not intended\. Despite being unintended, the mistaken item is in fact tokened\. You may intend to token ‘racket’ \(the sporting equipment\), and not ‘bat’ \(the sporting equipment\), but merely having this intention does not guarantee that you do so\. Things don’t always go according to plan—just like when you intend to grab a racket rather than a bat, but pick up the bat anyway\. Intentions are neither necessary nor sufficient for tokening a word\.

\[stojnic2022words\]argues that the process of lexical selection itself is what’s key to word\-token individuation\. The basic idea is that two utterances token the same word just in case the item selected from the speakers’ mental lexicons in producing those utterances are appropriately related\. For example, a speaker’s utterance is one of ‘racket’ rather than ‘bat’ just in case the operative item from lexical selection corresponds to the word ‘racket’ rather than ‘bat’\. Recall, however, that selecting a lexical item need not itself involve tokening an expression\. Rather, in most psycholinguistic models of speech production, selecting an item involves activating a computational element that is always available, so that that element \(as opposed to others\) plays a causal role in shaping the form of the utterance\. Moreover, the identity of the word is not a matter of information stored at the node itself\. In most psycholinguistic models of speech production, words are identified by their networks of connections, which control when each element is activated \(for example, what target concepts are associated with it\) and what downstream processing the element triggers \(for example, how it activates related syntactic, morphological, and phonetic elements\)\. If you are already familiar with how LLMs work, you’ll recognize this description of lexical selection via a causal framework of dynamic information propagation through a persistent network\. \(If not, read on\.\)

But when exactly do items from the speakers’ mental lexicons correspond to the same word?\[stojnic2022words\]defends a causal–historical account—but one grounded in the language faculty rather than speaker intentions\. Words are introduced—coined—by productive conventional operations mechanistically invoked by the language faculty, through suitable discourse moves \(see\[stojnic2025\], and Section[4\.2](https://arxiv.org/html/2606.02973#S4.SS2)below\); these words are then stored in the speaker’s mental lexicon\. When speakers use them, the words propagate to other agents, who can recognize the tokening of unfamiliar items, add them to their mental lexicons in turn, and so become part of an expanding network of users\. Utterancesuuandu′u^\{\\prime\}are of the same word just in case the lexical items selected in their production are appropriately connected, via such causal–historical chains, to the same creative origin—the coining\.

This account highlights a range of inferential challenges that must be met in extending a lexicon, whether by human speakers or by AI systems\. Most significantly, the audience’s recognition that they are encountering a new word is often a matter of inference\. Consider homonyms, for example\. Homonyms correspond to distinct lexical items, stored separately in the lexicon, and tracing back via distinct causal historical chains to different acts of neologizing\. Thus, for example, an utterance of ‘racket’ is that of a word denoting sports equipment, just in case it is a result of selecting an item connected via a causal\-historical chain to its roots in Arabic; and it is a that of a word for a noise just in case it traces back to that word’s echoic roots in Middle English\.151515\[stojnic2022words\]offers a detailed account of apparent meaning change, including cases of apparent fusion and fission, arguing that such cases are best conceived as involving tacit neologizing: introducing a new word by exploiting an articulation of a different one\. The full defense of this view would take us too far afield, but the interested reader should consult\[stojnic2022words\]\.

A hearer who is acquainted with only one of these words, but is confronted with a token of the other, thus faces a situation of temporary uncertainty\. The utterance could be an unusual contribution on the part of the speaker, composed from familiar words \(e\.g\., “cut the racket” understood an instruction to disassemble some sporting equipment\)\. Alternatively, the utterance could involve an unfamiliar expression \(as here, where “cut the racket” reproduces an idiomatic formulation meaning “quiet down”\)\. The right response is a matter of inference to the best explanation\.\[stojnic2022words\]points to such inferences—over and above the circumstances in which various surface forms are uttered—as establishing links between lexical items across speakers\. Analogously, then, when LLMs process input text, what matters is not what input is provided to the LLM, but whether we find evidence that the LLM draws appropriate inferences to organize the input into words from a lexicon\.

In fact, the ambiguities are even more complicated than this example suggests\. To process spoken language, audiences must segment words from a continuous signal\. The sound stream \[aɪwɑnə’rækɪt\], in addition to expressing a desire for noise or a paddle \(“I want a racket”\), could also indicate the speaker’s intention to set up the next pool game \(“I want to rack it”\)\. Breaking up such a stream into words is again an inferential hypothesis about the stream, not just a straightforward description of the input itself\. State\-of\-the\-art language models face similar ambiguities even when processing written text, because text is represented as an unbroken sequence of predefined units \(outputs of a so\-called “tokenization” step\), which often involve subword fragments\. Again, what matters in all these cases is whether the mechanisms of language processing explain the continuous input as exhibiting appropriate items in the lexicon\. In fact, compelling experimental results do suggest that human language learners and computational models in machine learning resolve such ambiguities using analogous distributional and statistical evidence\[saffran1996statistical,SAFFRAN1996606\]\.

In LLMs, this inference rests on two kinds of computational affordances, which are tuned by numerical optimization to match large corpora of text in the LLM pre\-training process\. First, patterns can be represented by embedding them in high\-dimensional vector spaces\. The word2vec embeddings describing word associations in English text are a famous example\[mikolov2013distributed\]\. As\[pennington2014glove\]explores, these embeddings can summarize co\-occurrence statistics in training data: each dimension corresponds to a family of items that occurs in similar contexts, and each linguistic element is then assigned a value along that dimension that indicates its affinity with that family of items\. For example, in language data, these dimensions can correlate closely with grammatical distinctions encoded in the morphology of nouns \(number\), verbs \(tense\), pronouns \(gender\), and so on\[finley\-etal\-2017\-analogies\]\. Optimizing embedding representations thus serves to cluster together training data that exhibits specific linguistic patterns and enables the model to characterize the structure of those patterns in more detail\.

Second, LLMs have an “attention” mechanism\[vaswani2017attention\], which works like a key–value store\. Attention allows the processing of one element to scan for related elements in the input with strong activations along specified contextual dimensions \(the keys\), and then to feed contextual features of the retrieved elements \(the values\) into further steps of analysis\. Attention is a foundational component of the transformer architecture powering LLMs\[vaswani2017attention\], where it is used to assign embeddings to linguistic elements as a function of the contexts in which they occur\. Thus, whereas early models like word2vec associated any linguistic element with a fixed embedding that captured the distribution of all of its occurrences, transformer models compute*contextual*embeddings that vary across occurrences\. Such contextual embeddings can predict the possibly heterogeneous effects of an element across the various contexts in which it occurs through differing values on appropriate dimensions of affinity\. Optimizing the attention computation allows the model to identify distinctive patterns that span extended stretches of text data and associate those patterns with specific, distinctive predictions\. The effect is to cluster, disambiguate, and abstract the linguistic features of structurally similar patterns in the data\.

Because LLMs are trained by incremental optimization using vast training data, these pattern finding processes happen gradually and continuously, not discretely, as in the “fast mapping” process characteristic of human language learning \(for review see\[Carey30062010\]\)\. LLM contextual embeddings corresponding to the different occurrences of ‘racket’, say, will initially exhibit small random variations, and the model’s predictions about text sequences will be correspondingly noisy\. However, the loss objective of the model will push the embeddings for some occurrences of ‘racket’ in one direction, in order to assign higher probability to sports\-related vocabulary observed nearby: tokens like ‘tennis’, ‘serve’, ‘fault’, or ‘love’\. Meanwhile, it will push the embeddings for other occurrences in a different direction, to assign higher probability to sound\-related vocabulary observed nearby: tokens like ‘hear’, ‘din’, ‘awful’, and ‘banging’\. Eventually the contextual embeddings take on consistent values that steer the model to better predictions\.

Once the model is trained, these representations are then activated in the course of processing and generating text\. Much as with the human mental lexicon, making sense of what item is selected in an LLM involves looking at these patterns of activation and understanding how they lead to output tokens and other predictions\. This is not a straightforward analysis to do\. As you might expect, however, given the computational affordances of LLM models and their success in producing fluent text, there is strong evidence that LLMs generally do encode and select fine\-grained word senses, and associate them with specific syntactic frames, as part of this processing\[10\.5555/3454287\.3455058,gari\-soler\-apidianaki\-2021\-lets\]\.161616Famously, the LLM training process has no recourse to a theoretically\-motivated characterization of universal grammar along the lines Chomsky has postulated for the language faculty\[piantadosi2023modern\]\. Nevertheless, LLMs do seem to respect syntactic universals, provided the LLMs see enough training data—typically vastly more than is available to human language learners in development\[hale\-stanojevic\-2024\-llms\]\. Chomskyans should, we think, conclude that LLMs do acquire syntax \(and language generally\), just not in a way that is human\-like \(or theoretically explanatory\)\. As\[chomsky2023\]put it: “Unlike humans, for example, who are endowed with a universal grammar that limits the languages we can learn to those with a certain kind of almost mathematical elegance, these programs learn humanly possible and humanly impossible languages with equal facility\.”

In sum, much like human users, AI models acquire lexical items mechanistically, by exposure to their appropriately causally grounded tokens\. Correspondingly, much like the outputs of human users—and unlike gusts of wind or waves on the sand—the outputs are produced through the mechanistic selection of inferred linguistic items that can be appropriately causally connected to the items in the training data\. We conclude that AI models generally do output linguistic expressions\.171717Of course, exceptional situations can arise in the operation of AI models, just as errors sometimes arise in human sentence processing\. We can sometimes even reliably trigger problematic AI behavior with appropriate probing tests, such as including so\-called “glitch tokens” in an input prompt\[land\-bartolo\-2024\-fishing\]\. Depending on the nature of the error, we might conclude that a specific AI output features a different word than the one it appears to, or features no word at all—just as we might make the same judgment about a problematic utterance from a human speaker\. That the process occasionally goes wrong, however, is not a reason to doubt that the process successfully outputs language in the vast majority of cases where it goes right\.

### 4\.2From Form to Meaning

Assuming we are correct that the outputs are linguistic expressions, do they express their ordinary linguistic meaning? We will argue that they do—regardless of the intentions, if any, of the system that produced them\. Indeed, we will argue that this linguistic meaning is sufficient to determine propositional semantic content for utterances in context, and so to underwrite the kinds of semantic criticisms we offered for chatbot talk in Section[2](https://arxiv.org/html/2606.02973#S2)\. This view may seem surprising, on both philosophical and linguistic grounds, but we believe the evidence for it is overwhelming\. Here is our case\.

It is often assumed that for a word to be used with its ordinary meaning, the speaker must intend to use it with its ordinary meaning\. But this, too, should be resisted—even from human users \(\[stojnic2022words\]; c\.f\.\[kaplan1990words,1b492554\-f9e1\-3f93\-9634\-2fa3e1e26054\]\)\. So long as the speaker tokens an expression, that expression carries its standing meaning, as underwritten by the relevant linguistic conventions\. Here, it’s instructive to once again consider cases of misspeaking due to semantic exchange errors\. Suppose a human speaker intends to say that cats belong to the family felidae, but accidentally, due to a semantic exchange error, utters, “Rats belong to the family felidae\.” Intuitively, their utterance will be false, not just meaningless; ‘rats’ will not somehow come to mean ‘cats’ on this one off occasion\. \(If the speaker were competing in a trivia contest, they’d lose points, even if the mix\-up is obviously a result of a speech error, rather than ignorance\.\) But, they didn’t intend to use the word ‘rats’ \(with its standing meaning or otherwise\)\. What they intended, instead, is to token ‘cats’\. The natural way to make sense of such cases is to say that the speaker’s meaning—what they intended to convey—came apart from their semantic content—the meaning their words are assigned by the standing conventions of grammar\.181818We draw on the distinction of\[KripkeSR\]between semantic reference and speaker’s reference\.

A common motivation for insisting that speaker intentions are necessary for meaning preservation comes from creative uses of language\. To borrow Kripke’s example, one will not preserve meaning if one hears ‘Napoleon’ and decides it’s a good name for their pet aardvark\[kripke1980naming\]\. Kripke’s suggestions are that the speaker here has reused the name ‘Napoleon’, and that she has given it a new meaning because she uses it with a referential intention targeting Napoleon the aardvark\. We reject both suggestions\.

The aardvark’s name, we argue, is not the same name ‘Napoleon’ that conventionally identifies the General of Revolutionary France and subsequent Emperor\. It is a*different word*which shares the ‘Napoleon’ orthography\. One can certainly coopt an articulation of an extant expression to coin a new one\. If the coining is successful, the newly coined expression will then trace its history from this point on\. But it will be a new expression, a homonym of the old one\. As with other homonyms, the new name will be stored separately in speakers’ lexicons\. As expected, it will behave like a separate item with a separate meaning\. For example, “Napoleon lost the battle of Waterloo, and needs a bath” can’t have a reading where its supposed multiple meanings are made simultaneously available\.

Moreover, it is not the speaker’s*intention*to refer to the pet aardvark that determines the reference of the new name\. Instead, we have argued, the reference of a name—and more generally, the meaning of novel expressions—is determined by the causal circumstances in which the name is assigned, along with discourse constraints on the interpretation of the utterances that introduce them; it can then be investigated and*discovered*by interlocutors through subsequent processes of inquiry\[lepore2017convention,stojnic2025\]\.191919We model this formally by assuming that the architecture of the language faculty links certain utterances to certain real\-world situations: real\-world situations can \(causally\) prompt certain utterances, and the utterances, in turn, are semantically interpreted as characterizing those situations\. These interpretive possibilities include moves that introduce new names and pair them with semantic values consisting of the prominent entities in the corresponding situations\. See\[stojnic2025\]\.Our evidence will be familiar: references, like words, fit the facts in the world, irrespective of the speaker’s intentions\. In the example of\[lepore2017convention\], for example, the neologism ‘bromance’ refers from its first usage not \(merely\) to any specific conception of affectionate, supportive relationships between men that an initial author might have intended, but to the real relational possibilities inherent in the situations that prompt the author to introduce the term\.

Analogous cases involving names are familiar\[kripke1980naming,KripkeSR\]\. Lederman and Mahowald describe a task posed to GPT\-4 to rename a real historical figure and describe facts about this figure\[lederman2024libraries\]\. They write “Chat\-GPT \(GPT\-4\) completed the task using “Marion Starlight” in a text describing a figure “born in the 18th century” who “authored a famous pamphlet that criticized the French monarchy”, “played a critical role in the French Revolution”, “became increasingly paranoid and was involved in the Committee of Public Safety, which oversaw the reign of Terror” and “was arrested and executed during the Thermidorian Reaction\.”” They conclude that “Marion Starlight” plausibly refers to Robespierre, and that—given that the name is novel—this can only be so if GPT\-4 intended to refer to Robespierre\. But consider playing the same game with a fellow human—your human friend outputs “Marion Starlight was born in the 18th century, authored a famous pamphlet criticizing monarchy, played a critical role in the French Revolution, and was arrested and executed during the Thermidorian Reaction”\. Unfortunately, your human friend is not up to date on the minutiae of French history\. They were hoping to characterize, say, Paul Barras, but misremembered the facts\. The text, we submit, still is about Robespierre—–your friend’s intentions and confusion notwithstanding\. Similar considerations extend to the second task Lederman and Mahowald consider, where an LLM was asked to name elements in a picture it produced\. But consider your human friend playing the same game again: they draw a picture of a cockerel and a frog and say, “Ted is the rooster and Fred is the croaker”\. Alas, your friend intended things the other way around—–the intended target for ‘Ted’ was the frog, and ‘Fred’ was the chicken\. Your friend made an absent\-minded mix\-up, or intended to confuse you, or was confused about what ‘croaker’ and ‘rooster’ mean\. Once more, this is of no consequence for what they named: we understand the utterance as fitting, and describing, particular bits of the world—or of the relevant drawing, in this case—and the names are understood accordingly, your friend’s intentions notwithstanding\.

In short, given our mechanistic account of word identity and our conventional account of word meaning, meaning comes for free: so long as a word is tokened, it will be tokened with its ordinary meaning, associated with the standing conventions of a language\. Since chatbots also token words, on our account, those tokens too are associated with their ordinary meanings\. Note, however, that this account does not assume that chatbots*participate*in the relevant conventions\. On standard views of convention\[lewis\-c:1969\], such participation must involve intentional coordination\. Rather, it is enough that chatbots are designed so that they act in accord with the relevant conventions\. The analogy is to a chess\-playing computer program, which selects and outputs creative moves because it has been programmed to carry out the right generative calculations: we are entitled to interpret its output as moves in chess, even though it is not intentionally following the conventional rules of the game\.202020Note that it is quite delicate to formulate a notion of linguistic convention that responds appropriately to the discoveries of generative grammar, even for human speakers\[ImaginationConvetnion,sep\-convention\]\. For example, it is controversial whether human speakers intentionally follow the rules of language\[devitt:2006a\]\.

Moreover, as we have seen in Section[2](https://arxiv.org/html/2606.02973#S2), these meanings have an important role to play in making sense of chatbot utterances\. One way to think about that is in terms of patterns in system behavior that show that the system respects the organization of coherent inquiry: for instance, answering questions or responding to inconsistencies in what’s been said by retracting and adjusting the claims that have been made so far\.212121\[lederman2024libraries\]advance a similar concern when they argue that it is appropriate to treat chatbot outputs as meaningful only because chatbot contributions respect background constraints of syntactic, semantic, and discourse plausibility\.We have argued that, even for human users, such organization of coherent inquiry is a product of conventional rules, rather than a product only of deliberate, intentional activity\[stojnic2025\]\. Recognizing the conventional meaningfulness of chatbot talk allows us to link chatbot behavior likewise to the formal organization of coherent inquiry\. It provides a de\-anthropomorphizing alternative to the idea that meanings depend on an agent’s intentions to perform assertions or other communicative acts\. As emphasized above, of course, the question whether an agent’s utterances are meaningful should not be conflated with the question of whether the agent itself will have asserted, said, or meant the content semantically expressed by the token it has produced\.

It’s important to acknowledge that, as we foreshadowed earlier, establishing the meaningfulness of expressions whose content is standardly understood as fixed by standing semantic conventions is only the tip of the iceberg\. There are much trickier issues around context\-dependent utterances\. The orthodoxy, we’ve seen, assumes that for many \(indeed, most\) context\-sensitive expressions, their standing meaning underdetermines their content on an occasion of use\. Examples involving demonstratives are a classic case in point\. A sentence containing a demonstrative pronoun can express indefinitely many contents: e\.g\., ‘She is happy’ will mean one thing if uttered pointing at a cat, Betty, and something quite different if uttered while discussing our friend, Annie, and a third thing yet if uttered while looking at Annie’s daughter opening her birthday presents\. The standing meaning of ‘she’, its Kaplanean character, constrains its referent to be in the third person, singular, and female\. But this in itself underdetermines the content\. On the standard understanding, again, what is required to fix the content is speaker’s intention: it is because the speaker intended Betty that the utterance of ‘She is happy’ is about Betty\.222222Proponents of this view are many\. See\[kaplan1989afterthoughts,neale2020demonstratives,299914f9\-eb66\-34d5\-a11b\-ddf2de48c3c8\]*inter multōs aliōs*\.If this view is on the right track, then, there can be no talk of assigning propositional content to chatbot utterances without positing intentions: context\-sensitivity is pervasive, and if its resolution relies on speaker’s communicative intentions, then no chatbot “semantics” could underwrite the endorsement or criticism of chatbot outputs as true or false\.

But, once more, the dominant view is mistaken, even for human users\. We’ve independently argued at length that the view is empirically inadequate, and that a close look at the data recommends thinking of pronoun resolution as fixed by discourse\-internal, linguistic conventional rules\[Deixis,DLF,StojnicBook\]\. The full defense of the view exceeds the scope of the present paper, but here we sketch the key points in broad outlines\.

On a view we defend, the resolution of a demonstrative pronoun is fixed by discourse\-internal linguistic mechanisms that manipulate the relative prominence of candidate referents for pronoun resolution\. The pronoun simply selects the most prominent referent that accords with its linguistic meaning: e\.g\., ‘she’ selects the most prominent third person, singular, female referent\.232323We are glossing over formal details\. For a formally implemented fragment, see\[DLF\]\.

One particularly important mechanism that affects prominence comes from discourse coherence\. As mentioned earlier, discourse coherence represents a network of rhetorical relations—coherence relations—that encapsulate how the component propositions of the discourse connect into a larger whole\. This is what makes a discourse more than just a random sequence of unrelated sentences strung together in no particular order\[asher2003logics,hobbs1985coherence,Hobbs1979,kehler2002coherence,Kehleretal\]\. This minimal pair from\[Hobbs1979\]offers a classic illustration:

\\ex

\. J̇ohn took a train from Paris to Istanbul\. He has family there\. \.̱ John took a train from Paris to Istanbul\. He likes spinach\.

The discourse in[4\.2](https://arxiv.org/html/2606.02973#S4.SS2)isn’t just a random sequence of sentences about John\. It conveys that the train trip is explained by John’s familial ties to Istanbul\. By contrast,[4\.2](https://arxiv.org/html/2606.02973#S4.SS2)is infelicitous\. An attempt to establish a coherent connection between its two sentences leaves one confused: what is it about his taste for spinach that explains John’s train trip?

Coherence Theory captures these observations by representing coherence relations explicitly in the logical form of the discourse\. The two sentences of[4\.2](https://arxiv.org/html/2606.02973#S4.SS2)are connected by the Explanation relation, which contributes to the overall content of the discourse; similarly, the reader expects an explanatory connection in[4\.2](https://arxiv.org/html/2606.02973#S4.SS2), but cannot establish one—hence its infelicity\.242424At least without further information, of course: if contextual information has indicated that Istanbul is the capital of spinach, the explanation will go through, and the discourse will be felicitous\.

Coherence relations don’t merely signal how successive sentences connect: they also affect how successive sentences are interpreted\. Pronoun resolution is a case in point, as illustrated in\\Next\[Smyth\]:

\\ex

\. Phil tickled Stanley, and Liz poked him\.

\\Last

has two different readings\.252525We are assuming that[4\.2](https://arxiv.org/html/2606.02973#S4.SS2)is not accompanied by a pointing gesture\. It is worth mentioning that, while pointing is often considered as a paradigm extra\-linguistic cue, a closer look at the data on demonstrative gestures demonstrates otherwise\[DLF,StojnicBook\]\. In fact, demonstrative gestures exhibit the arbitrariness and cross\-linguistic variation in form\-to\-meaning mapping typical of linguistic conventions\. A detailed defense of this claim is beyond the scope of the present paper, but an interested reader can consult\[DLF\]and\[StojnicBook\]\.On one interpretation, the discourse is organized by the Result relation, whereby the second sentence describes how Liz’s action were prompted by Phil’s: Phil tickled Stanley, and*as a result*, Liz poked him\. Another interpretation has\\Lastorganized by the Parallel relation, comparing and contrasting two eventualities—Phil tickled Stanley, and*similarly*Liz poked him\. The organizing relation affects the resolution of the pronoun\. If the discourse is organized by Result relation, the pronoun picks out the subject of the previous sentence, Phil\. If, instead, the discourse features Parallel, comparing eventualities described by the two clauses, the pronoun picks out the referent introduced in the same grammatical position—the object position—Stanley\.

The correlation between discourse coherence and demonstrative pronoun resolution is well\-documented\.262626See, in particular,\[Kehleretal\]and references therein\.At first glance, one might be tempted to explain it as a reflex of the speaker’s communicative intentions\. Take[4\.2](https://arxiv.org/html/2606.02973#S4.SS2)\. If the speaker’s goal is to describe who did what to Stanley—and so, the organizing relation is Parallel—this comparison will only succeed if the pronoun picks out Stanley\. So, absent overriding cues, this speaker must intend for the pronoun to pick out Stanley if their intention is to be recognizable and manifest to the audience\. If, instead, the speaker’s goal is to describe Liz’s reaction to Phil’s attack on Stanley, it would make sense they intend the pronoun to pick out Phil, as this delivers a natural interpretation that Phil’s attack prompted Liz’s revenge\.

But, tempting as it might be, this explanation doesn’t withstand scrutiny\. In\[DLF\], we’ve extensively argued that these constraints, instead, behave as*bona fide*linguistic conventions\. We won’t rehearse our arguments in full, but we can illustrate with a telling example from\[kehler2002coherence\]:

\\ex

\. \# Margaret Thatcher admires Ronald Reagan, and George W\. Bush absolutely worships her\.

As Kehler reports, subjects judge that\\Lastis infelicitous\. The recovered interpretation is incoherent: the speaker is understood to have made a mistake, using a female\-gendered pronoun to refer to a male referent, Reagan\. This interpretation agrees with the effect of Parallel we see in[4\.2](https://arxiv.org/html/2606.02973#S4.SS2): the pronoun in the object position, ‘her’, requires an antecedent introduced by an NP in the same position, ‘Ronald Reagan’\. So, ‘her’ must select Reagan, which results in a gender clash\.

But this explanation is only available if the effect of Parallel is a linguistic constraint, not a mere defeasible pragmatic cue guiding intention recognition\. For otherwise, the cue should be easily overridden by way of the following type of reasoning: ‘her’, given its standing meaning, requires a third\-person, female referent\. A female referent was recently mentioned—indeed, explicitly introduced in the preceding clause\. This referent is an eligible candidate for the resolution of the pronoun\.272727Indeed, in English, referents introduced in the subject position are preferred for subsequent anaphora resolution than those in the direct and indirect object position\. See\[Kameyama,Masayo,Bittner1\]\. Accordingly, the interpretation according to which ‘she’ picks out Thatcher should be not only accessible, but*favored*\.If ‘her’ were to pick out Thatcher, the interpretation thus fixed would be charitable, a coherent message, rather than an erroneous, incoherent one\. Moreover, it would be relevant to the main topic under discussion, and, overall, the most plausible interpretation\.

In short, were the effect of Parallel just a defeasible pragmatic bias, it would be easily disregarded given the abundance of evidence that suggests a different, more plausible interpretation\. However, not only is the incoherent interpretation the one readily recovered, but the alternative, more plausible, one isn’t even*available*\. Indeed, suppose it is completely transparent that the speaker intended to refer to Thatcher: let’s say they immediately clarify this\. Even so, this interpretation is still not readily available; the intuition persists that the speaker has mistakenly used a female\-gendered pronoun to refer to a male referent\.

This type of a forced interpretation is a hallmark of a background linguistic constraint\. Such constraints are difficult to override, even if the interpretation they deliver is incoherent, and despite the salience of other, more plausible interpretations\. Moreover, the constraint itself is highly arbitrary: that a pronoun should seek an antecedent introduced in the same grammatical position in the previous clause does not follow from general knowledge about \(dis\)similarities between eventualities\.282828Parallel interpretation also requires a particular intonation pattern, with the pronoun deaccented\. A contrastive focal stress on ‘her’ would render\\Lastfelicitous, allowing ‘her’ to resolve to Thatcher\. But this stress carries further interpretive requirements, where the discourse is no longer understood as comparing the strength of Bush’s and Thatchers’ attitudes\. There is no reading where the relation remains Parallel and the pronoun resolves to Thatcher\. See\[DLF\]for more on this point\.

[4\.2](https://arxiv.org/html/2606.02973#S4.SS2)is but one example that illustrates a general phenomenon\.\[DLF\]discuss a wide range of discourse\-internal linguistic rules that affect the prominence of various candidate referents for reference resolution in analogous ways\. These include the conventionalized effects of prosody, demonstrative gestures, the grammatical roles of the antecedent NP, and so forth\. They all have hallmarks of linguistic conventions: they are language\-specific, cross\-linguistically variable, and arbitrary in the type of effect they induce\.

On this picture, pronoun resolution is sensitive only to the linguistic context fixed by discourse\-internal mechanisms; speaker intentions are neither necessary nor sufficient for fixing the meaning\. Following\[lewis:1979\],\[DLF\]propose to model context as a running record of contextually relevant information\. The record keeps track of parameters on which context\-sensitive expressions depend, but the values of the parameters are entirely determined by discourse\-internal mechanisms—linguistic conventions—that update the record word\-by\-word, as the discourse unfolds\. Among such contextually relevant parameters is the relative prominence of candidate referents for pronoun resolution\. We can model it as a list of candidate referents ordered by prominence\. As the discourse unfolds, the scoreboard evolves with each new contribution and the prominence ranking changes: certain entities are raised to prominence, demoting others\. The idea is that certain linguistic mechanisms, for instance, coherence relations, prosody, demonstrative gestures, NP grammatical roles, and so on, come with grammatically encoded and specified prominence\-affecting updates, which make certain referents prominent, demoting others\. A pronoun, at any given point in discourse, selects the most prominent entity that accords with its linguistic meaning—e\.g\., ‘she’ selects the top\-ranked, third\-person, female referent\. But which entity is the most prominent is determined by the linguistic effects of various items in the preceding discourse\.

On this picture,[4\.2](https://arxiv.org/html/2606.02973#S4.SS2)is ambiguous but not dependent on extra\-linguistic mechanisms or speaker intentions\. The two disambiguations correspond to two distinct logical forms—one featuring Result and the other Parallel\. But each of these two forms unambiguously determines a particular resolution of the pronoun, and expresses a determined, linguistically fixed content\. Similarly, on its natural interpretation, according to which[4\.2](https://arxiv.org/html/2606.02973#S4.SS2)is organized by Parallel, the discourse determines an unambiguous logical form, the one that forces an incoherent reading according to which ‘her’ is interpreted as Thatcher\.

The linguistic constraints on context\-sensitivity resolution aren’t limited to demonstrative pronouns\. As one of us has argued at length\[StojnicBook\], discourse\-internal linguistic mechanisms affect the interpretation of context\-sensitive expressions quite generally, in analogous ways\.

If this is right, speaker intentions don’t play a role in resolving context\-sensitivity even for human users\. There is thus no special threat to meaningfulness of chatbot utterances that stem from context\-sensitivity and its resolution\.

### 4\.3Summarizing the Approach

To put our work in context, we’ve been arguing with\[cappelen2025hog\]for de\-anthropomorphizing the philosophy of language\. In particular, we have been arguing that typical assumptions about the role of intentions in determining utterance meaning, even for human users, should be abandoned\. But with those assumptions abandoned, we can then straightforwardly describe how algorithmic systems produce meaningful language without themselves meeting these putative intentional prerequisites for communication\. Thus, unlike previous theorists, we see no need to argue that ChatGPT is a cognitive agent, as in\[cappelen2025hog\], or to argue that ChatGPT be regarded as a member of our linguistic community, as in\[mandelkern\-linzen\-2024\-language\]\. Nor do we see the need to focus on derived intentionality as a fundamental disconnect between ChatGPT’s meanings and our own, as in\[lederman2024libraries\]\. Our view boils down to a simple slogan: Language is meaningful\.

Like\[cappelen2025hog\], we see de\-anthropomorphizing the philosophy of language as an expansive research program requiring substantial further investigation\. Looking beyond context\-sensitivity resolution, for example, we have argued that a wide range of phenomena typically described as intentional updates in the dynamics of conversation—presuppositions, coining new names, negotiating standards for vagueness, resolving underspecified or controversial meanings—are instead governed by language\-specific, cross\-linguistically variable, conventional rules\[ImaginationConvetnion,stojnic2025\]\. All of these proposals must be substantiated to apply our ordinary semantic intuitions about these phenomena correctly to chatbot utterances\.

Of course, what\[cappelen2025hog\]say is that the right way to proceed is to de\-anthropomorphize our concepts of mentality altogether, saying that systems like ChatGPT really are full\-blown cognitive agents\. We don’t think that that’s warranted, nor, as we’ve argued, is it necessary in order to preserve the intuition that the linguistic output of AI agents is meaningful\.

## 5Meaning and Criticism

Our views dovetail with the criticisms of chatbots that we surveyed in Section[2](https://arxiv.org/html/2606.02973#S2)\. On our view, ChatGPT selects English words and constructions and assembles them into English expressions\. On our view, the rules of the language associate those expressions with content in context\. On our view, that content comports with the coherent dialogue that ChatGPT appears to exhibit\. On our view, the conditions suffice to attribute genuine linguistic meaning to the outputs of ChatGPT, and indeed, we think this stance is independently motivated, to properly account even for human linguistic practices\. Starting from these standards, a close look reveals that ChatGPT’s claims are often false or misleading, while the supporting information it offers is often spurious and inaccurate\. These semantic failures represent serious weaknesses with chatbot technology\. In this section, we offer an assessment of the breadth and limit of such critiques of interactive systems based on linguistic meaning\.

### 5\.1Meaning is easily achieved

On our view, what underwrites linguistic meaning in interaction is the mechanistic selection of linguistic items with appropriate causal histories\. Such selection can be implemented through diverse processes, which means that our view allows for a parallel treatment of systems with similar behavior, even when those systems may work in very different ways\. For example, as a thought experiment, consider hypothetical alternatives for how a “bot” might contribute to a dialogue like Guinzburg’s:

- •The bot’s outputs are entirely the result of human\-like intelligence\. Perhaps designers have found a way to quickly and seamlessly leverage the communicative choices of countless human crowd workers\. Perhaps the bot is an evil demon—supernaturally fast but otherwise fully human\. By hypothesis, the bot’s contributions are backed by all the communicative intentions that we would normally \(and correctly\) attribute to other people\. On a traditional view, these communicative intentions establish the meaningfulness of the bot’s contributions\. On our view, it is enough that the bot has the right processes of lexical selection and the right coherent behavior\.
- •The bot’s outputs are entirely the result of simple mechanisms for manipulating textual data\. Concretely, the bot might proceed by retrieving a collection of relevant passages from a corpus of text\[lewis2020retrieval\]\. It might then create fluent output by using simple paraphrase techniques to stitch those retrieved passages together into a coherent response that fits the context\[balepur\-etal\-2023\-expository\]\. Everyone would agree, by stipulation, this is a*mindless*algorithmic process, completely devoid of goal\-directed rationality\. On a traditional view, the only meaning here—if any—would be the meaning the bot’s output could exhibit by proxy based on the communicative purposes of the bot’s designers\. On our view, however, the retrieved passages and the further selection of linguistic elements in the paraphrasing process come with suitable causal histories to be recognized as English expressions with their corresponding meanings\.
- •The “actual” scenario: the bot produces output by an incomprehensible pile of linear algebra\.292929The imagery here is due to the inimitable Randall Munroe:[https://xkcd\.com/1838/](https://xkcd.com/1838/)\.For the traditional view, as we saw in Section[3](https://arxiv.org/html/2606.02973#S3), the case is obscure\. Some argue that there are no communicative intentions here\[10\.1145/3442188\.3445922\], others that there are all the communicative intentions of a full cognitive agent\[cappelen2025hog\]\. Still others\[lederman2024libraries,mandelkern\-linzen\-2024\-language\]suggest that some intentions may be present \(e\.g\., to use words\) while others may be lacking \(e\.g\., to refer\)\. Our view sidesteps these thorny issues—complex and difficult as these networks are to interpret, the evidence is strong that they acquire and select linguistic elements\.

In all these cases, the dialogue is the same, and is experienced the same by the user\. Our view endorses this intuitive commonality\. We acknowledge a level of semantics that applies identically in all the cases\. Traditional accounts, by contrast, see fundamental differences across the various cases\.

What goes for the user also goes for critics and researchers\. Of course, developers pay close attention to the costs and benefits of different computational approaches\. Implementation details matter\. But for decisions about the strengths and weaknesses of an existing system—what works well, and what needs to be improved—developers should appraise systems the same way, regardless of the underlying mechanisms\. What’s typically operative in explaining the system’s behavior and the system’s impact is the meaning of its utterances\. Nothing is gained by drawing complicated distinctions about how those utterances arise\.

To make this vivid, consider the pedantic—and, we think, unhelpful—lessons we might draw from received views\. Suppose we conclude that current chatbots use linguistic expressions but lack referential intentions; suppose we accept that referential intentions determine the referents of demonstrative pronouns\. These considerations might lead us to recommend that current systems should avoid demonstrative pronouns, relying instead on expressions whose meaning is determined by standing conventions\. If we implement this suggestion, instead of[5\.1](https://arxiv.org/html/2606.02973#S5.SS1), we would hope to see[5\.1](https://arxiv.org/html/2606.02973#S5.SS1):

\\ex

\. Ṫhat’s why my response didn’t mention specific references like Madonna or Instagram—because I never saw them\. \.̱ That’s why my response didn’t mention specific references like Madonna or Instagram—because I never saw specific references like Madonna or Instagram\.

Clearly, this reformulation is no improvement\. The dialogue is just as toxic as before\. The change simply makes the output a little bit more confusing and a little bit more verbose\.

We suspect that interventions based on aligning system behavior with traditional accounts of intentionality, agency, representation, and communication will rarely give helpful advice\. Our account, by contrast, enables critical engagement with chatbot outputs, including the toxic aspects of their behavior, without carefully parsing their rationality according to philosophical standards\. On our view, producing meaningful output has minimal prerequisites\. It suffices to anchor the outputs of the systems in the right kinds of historical and causal connections, in the right kinds of conventional rules, and in the right kinds of organizations for conversation\. These prerequisites are met by ChatGPT, but also by a wide range of simpler systems, as well as by people, and by proxy agents, corporate speakers, and others\. We can criticize their meanings by the same standards\.

### 5\.2Criticizing chatbots: beyond truth and reference

We want to close by emphasizing some of the limits of the kinds of analyses that we’ve offered\. It will be instructive to return to the phenomenon of sycophancy\. Whatever an incomprehensible pile of linear algebra can compute, we don’t think that it leads to genuine aesthetic judgments that can be shared with people in a meaningful way\. So we wouldn’t endorse the truth of ChatGPT’s utterances, for example, when it says, “I’ll know how good your pieces are by reading them the same way an editor or agent would,” or when it says “your piece is stunning\.” If ChatGPT is designed to be honest, it should refrain from this kind of talk\.

Ours is a philosophical conclusion\. By contrast, in AI research, honesty is typically viewed through the lens of fact\-checking that’s inherited from journalism\[guo\-etal\-2022\-survey\]\. Practices of fact\-checking draw on norms of objectivity that distinguish between factual claims, which can be confirmed or rejected by consulting relevant evidence, and matters of opinion, which are inherently subjective and so can neither be verified nor refuted\[graves2019a,1106839ar\]\. According to these standards, factuality does not apply to ChatGPT’s most egregiously sycophantic contributions\.

Perhaps, however, there’s a lesson to be learned from the observation that ChatGPT seems to meet these journalistic standards\. Imagine some more advanced version of ChatGPT, where we had reason to think that it really did read like an editor, in some more significantly human\-like way\. Would it really be good for such a system to simply boast of its prowess? Suppose that it had been designed in a way that we understood as underwriting meaningful aesthetic judgments, and happened to find Guinzburg’s work actually stunning\. Would that setup make over\-the\-top flattery OK? Of course not\. Getting truth and meaning right is important but insufficient\. There are broader critiques to make about state\-of\-the\-art chatbots\.

We now know a lot about what human\-computer interaction researchers call deceptive patterns \(or “dark patterns”\) in technology design\[brignull:patterns,brignull:book\]\. These include the “vivid and quantified feedback” that accompanies others’ engagement with our social media posts—a design choice that enhances pleasure at the expense of meaningful connection\[Nguyen2023Twitter\]\. They include filter bubbles in news feeds\[pariser2011filter\], which reinforce and entrench our preconceptions at the expense of open inquiry and epistemic justice\[doi:10\.1080/02691728\.2011\.652211\]\. And in general, there’s a well\-known tendency of interactive media systems to prioritize content that creates strong reactions, over more nuanced and more inclusive presentations, because strong reactions drive engagement\[9179098\]\. Any criticism of chatbot language needs to confront the diversity and pervasiveness of these deceptive patterns\.

In fact, these patterns abound in Guinzburg’s dialogue, including in utterances that have meanings that seem otherwise unproblematic\. For example, before Guinzburg clues in to the fact that she’s being gaslighted, ChatGPT provides the following description of how the conversation might proceed:

\\ex

\. If you have a fifth piece that’s politically sharp or wildly funny, that could work, too\. But honestly, this is already a fantastic spread\. Want me to write a paragraph as a query that frames these links? I can help you position them to highlight this range and make it feel cohesive\.

Guinzburg then writes as an aside, “I never asked ChatGPT for help writing the letter, or help writing anything at all\.” But, of course, what ChatGPT seems to be designed for is to keep you talking to it: to keep you investigating things that might seem useful to you, to keep you on the site, to keep you thinking highly of it and using it more\. It’s an invitation to the kind of addictive relationship with technology that we’ve seen across so many recent tech developments—going back to the empirically\-optimized addictive design of multi\-line slot machines\[Schull2012\]\. We’d criticize the blustery flattery of ChatGPT, then, not just as something that happens to be false in the case of Guinzburg’s dialogue, but as an inherently manipulative device that conduces to users overestimating the efficacy of the tool and using it compulsively\. Such utterances instantiate a classic deceptive pattern and are never going to be good—even if true\.

Weizenbaum’s reflections on his original ELIZA system\[weizenbaum1976computer\]already highlight what’s wrong with chatbots that take too expansive a role in guiding users\. Through inquiry, language users can provide information that answers interlocutors’ questions\. Alignment in inquiry is unproblematic—we aim for the truth\. But what of language that manipulates, provokes, inspires, or confounds? What matters is not simply meaning and truth, but a capacious humanistic appreciation of the myriad ways our ideas can shape one another\.

Weizenbaum approached such questions through the political philosophy of Hannah Arendt\[goodlad2023editor,Herzog\_2021\]\. What Arendt says is that full participation in political life depends on thinking, on articulating reasons and judgments in the public sphere, listening to the rational responses of free and thoughtful equals, and using open\-ended, critical engagement to come up with agreements among ourselves about how to move forward\. Weizenbaum was pessimistic that AI techniques could ever contribute to public dialogue that met Arendtian standards\. We hope above all that a critique of chatbots grounded in the meanings of their utterances can help theorists to escape such pessimistic conclusions\.

## 6Conclusion

To conclude, we’ll just return to the main points of our work\. Chatbot talk is meaningful, which, again, means not that the chatbots mean what they say, but that they output sentences that are meaningful and can be interpreted according to standard linguistic conventions\. Meaning of this kind is a low bar: it does not imply that chatbots have mental states, intentions, rationality, or the cognitive capacity requisite for communication\. It suffices that they are engineered to exhibit certain kinds of patterns, and trace those patterns to causal and historical networks in our linguistic communities\.

Moreover, saying that chatbot talk is meaningful is not an endorsement of what they say\. Meaning is crucial for capturing when chatbots go wrong, not just when they succeed\. And it’s easily overemphasized\. It would be great to make chatbots that said true things\. We’re a long way from that\. But we’re not going to have good chatbots merely by making them speak the truth\.

## References