Summaries without originals
A recurring concern with LLMs is that tech companies will run out of data with which to train them and they will gradually become less and less useful. How would tech companies, which have perfected so many means of surveillance and data extraction, run out of data, especially when institutions like universities appear eager to sell even more to them than they can steal? The fear is that the content that LLMs produce will overwhelm all the spaces from which human-made content is currently being harvested, and the models will start being retrained on their own excrement. Instead of modeling what occurs in human discourse, the models would begin recursively modeling their own predictions at an ever further remove from contemporary human practice, eventually succumbing to what some researchers have called “model collapse,” a kind of mad cow disease for machines.
The models will then have become victims of their own usefulness. LLMs need data based on humans communicating with intentionality, but their output interferes with that kind of intentional communication and evokes the fear that LLMs will end up producing discourse that is processed only by other LLMs: the future in which there is “endless content generated by robots, enjoyed by no one, clogging up everything, and wasting everyone’s time,” as Lincoln Michel put it (and which I wrote about previously here and here).
But no matter how much AI-generated spam is spewed into our communication channels, it seems unlikely that humans will stop generating meaning or having intentions of their own. There has been an ongoing ideological campaign to convince us that interacting with machines is preferable to interacting with other people who have their own agendas (self-checkout is so convenient! chatbots are friends who really listen!), but that all underscores the increasing value of someone else’s attention.
Despite tech companies’ ongoing efforts, it’s not as if we’ve suddenly realized that expressing ourselves and listening to other people is nothing but a hassle in every conceivable situation and that it would be far more convenient to abandon our language skills altogether. By loosing LLMs on the world, tech companies haven’t persuaded humans to mute themselves and embrace the machine-readable utopia. Instead LLMs shift the spaces where meaning can be captured, and how positions within and around those spaces are distributed.
The models don’t just pollute the waters where tech companies once fished for their data; they also reroute circuits of attention and comprehension among those whose social relations are inflected by them. Whose voices matter and why, who is trying to achieve what, who pays attention to whom — LLMs offer an agile means for altering or reinforcing these arrangements, regardless of whether the content they produce is all that pertinent or is processed only by other machines. They can disrupt and reconfigure the interpersonal and social contexts necessary for meanings to be meaningful without being factually reliable themselves.
When LLMs are used to automate text, they also produce obscurity about who is speaking and why and what sort of effort it cost them. It imposes a lack of context on some people, making them communicate nonreciprocally with another party who masks their commitments by speaking through a machine or delegating to its haphazard capabilities. It’s not as though “human interaction” is a universal nuisance, as some of the more unhinged propaganda for automation will sometimes suggest. It’s that interactions are occasions for power struggles as well as communication; they are asymmetric, and the exchanges are always unequal in some dimension — whether in terms of the effort put in, or the attention paid, or the clarity of the takeaway, or the achievement of objectives.
LLMs are is not just word generators; they are weapons in these struggles. If you are made to engage with AI, it should probably be understood as a scam in progress, a punishment, or a stigma: To the powers that be, you didn’t warrant the recognition of another human being. You can make do with AI customer service, an AI tutor, an AI therapist, an AI boss. You can make your contributions through an LLM because no one is really interested in hearing your voice.
The implicit purpose of many AI applications will be to dehumanize others in communication situations, not to entertain or empower those who are compelled to use them. Empowerment accrues to those in a position to impose AI technology (and all its intrinsic biases) on others and not just to those who have some savvy way of using it themselves. In some cases LLMs can fill communicative space with placeholder content just to allow for more power struggles, more demonstrations of authority; the idea that they could produce something pertinent becomes an enabling pretense.
In a recent newsletter, Matt Levine pointed out that “large language models are not in the business of interpreting reality and summarizing it accurately; they’re in the business of completing texts in plausible ways. That is a risky approach to financial modeling but an incredibly efficient approach to summarizing meetings.” That is, they are not reliably informative but nonetheless useful in maintaining corporate hierarchies and differentiating who needs to be listened to and who needs to be present, and who can use technology to insulate themselves. Departing from an accurate account of “reality” is a small price to pay, and let’s not forget that the whole point of having power is to make your own reality.
***
When LLMs are promoted as something consumers would adopt voluntarily, it serves as an alibi for the technology’s punitive and exploitative functionality. Sometimes the promise is that generative AI will level the playing field, as with claims that the technology will “democratize” content creation or coding skills. But more typically the emphasis is on how it will save consumers effort — it will make you feel like a boss over some fully compliant servant who does the work for you.
Recently the technology’s supposed ability “to search the web for you” has been emphasized in newly launched services like Arc Search and Perplexity. These have been marketed as “AI assistant”-like technology that will ultimately save you the trouble of knowing how to do things and make decisions on your own (which I’ve also critiqued recently as a kind of masochism), but in practice they work more like ad blockers. They respond to a search query by using LLMs to summarize the content of some of the returned links and repackage that information in a few generated paragraphs on a newly fashioned page. That way, users don’t have to wade through the sponsored links and SEO-driven garbage that currently pollutes search engines in their terminally “enshittfied” condition.
Because search engines have become so adversarial, the LLM-driven summary tools appear fairly useful for now. The summary tools vividly contrast with the nightmare of LLM content making communication in general seem untrustworthy, rigged against you, liable to erupt anywhere to baffle, frustrate, or pander to you. They can promise to help you escape rhetorical predation and protect you from unwanted claims on your attention, rather than constituting those claims.
Summarization is such a plausible use case for consumer AI that political scientist Henry Farrell is able to analyze AI here in terms of who controls the “means of summarization.” He argues that there will be “ferocious fights between those who want to make money from the summaries, and those who fear that their livelihoods are being summarized out of existence,” a conflict that is already playing out in various intellectual property disputes he lists.
Farrell’s analysis pivots on the cannibalizing nature of AI summary and the looming threat of model collapse. LLMs rely on having original material to summarize, or else the “map will eat the territory.” But the tech companies developing AI are selling it as a replacement for “genuine human interaction,” conceiving of that production of original material as a cost that can be reduced, an inefficiency that can be eradicated. Farrell writes:
Like search engines, the summarizations that LLMs generate threaten to devour the territories they have been trained on (and both OpenAI and Google expect that they will devour traditional search too). They are increasingly substitutable for the texts and pictures that are represented. The output may be biased; it may not be able to represent some things that would come easily to a human writer, artist or photographer. But it will almost certainly be much, much cheaper. And over time, will their owners resist the temptation of tweaking them with reinforcement learning, so that their outputs skew systematically towards providing results that help promote corporate interests? The answer seems pretty obvious to me.
In other words, there is no reason to assume that companies won’t engineer summaries to be just as distorted by commercial incentives as search engines have become. And as many have pointed out, the summarizations are designed to seize the attention that would otherwise be directed to the originals and exploit it.
When AI companies extract the meaning from content without compensating its creators and make it possible for their clients to evade those costs too, they disincentivize the production of meaning and eventually starve their own models, which can’t produce anything meaningful on their own. So when those companies disregard copyright laws — which are theoretically designed to promote the content creation — and argue that those laws are a barrier to innovation, they are undermining the conditions of their own possibility. “If LLMs rely on reasonably high quality knowledge to keep on working,” Farrell points out of an Andreesson Horowitz claim about copyright laws stymieing AI development, “this is the exact opposite of true. The actual ‘bottom line’ is that declining to acknowledge the cost of producing such knowledge will either kill or significantly hamper the technology’s development.”
That seems right to me, but it also raises the question of whether summaries really are “increasingly substitutable for the texts and pictures represented,” and if so, what ideological conditions make it so. In what situations is that true, and what forces can make those situations more prevalent? What makes the idea of a summary a reasonable pretense for other kinds of appropriation and exploitation? Can you really extract what’s important about a piece of content with a summary, as though the original form of it was just an arbitrary container for a specific meaning? How do you paraphrase something without also interpreting it (and why would an LLM’s statistical method of “interpretation” be appropriate to any occasion)? Beyond the stakes of who controls the means of summarization are the stakes of the perceived usefulness of summaries themselves.
Farrell expedites his argument with the claim that “as these technologies get better, the summaries can increasingly substitute for the things they purportedly represent.” But that doesn’t seem self-evident. When he asks “do you really want to buy the 50,000 word book, if you can get the summary on the Internets for free or for cheap?” it is easy for a humanities-trained sap like me to counter that with an appeal to the value of form and style of some texts over and above the information they convey. It’s like asking, Why watch TV shows when you can read online recaps? It makes me want to break out Barthes’s The Pleasure of the Text and summarize it for you.
AI technologies don’t need to get better at summarizing for their to be a fight over their implications. They can get better at redistributing attention according to the prerogatives of those in a position to impose them without getting any better at summarizing anything, because the demand for them may not be driven by a desire for accurate information or productive efficiency but for power. The summaries don’t have to be accurate “maps” with respect to an underlying original; they just have to promulgate the conditions where some people get to skip the meetings that other people have to attend, where some people have to make the data that other people get to profit from.
Another way of putting this is that the summaries don’t have to summarize anything to be useful as a substitute for some other thing held to be valuable; they just have to seem capable of redistributing that value to whoever controls the summarizing tech. Farrell takes as given the “the general value” of summaries, that they are “usable maps” that steer us through “big inchoate bodies of information.” But when can be made spuriously and spontaneously, it matters less that they are usable maps and more that they demonstrate how the mapmaker effectively creates the territory, capable of investing it with significance through how they choose to attend to it. When the ostensible queries for knowledge only serve the purpose of calibrating power relations, it doesn’t matter whether they are effectively summarizing anything, because its “content” will be the relations it organizes, not the information it does or doesn’t contain. The problem is not just that LLMs might run out of stuff to summarize or disenfranchise the people who make the stuff being summarized, but that they will help organize a set of social relations in which summaries don’t need originals.