Look at a stone

One of the main assumptions behind large language models is that words are no different than numbers. All words can be reduced to numbers; all meanings can be assigned a definite value that is ultimately arbitrary. Meaning itself is a meaningless subjective mirage, something that can finally be eradicated once language is solved once and for all, its workings fully formulated and rendered calculable, like the operations of any other piece of machinery.

An underlying implication is language use is not free and unconstrained but predetermined and predictable; that is, the human use of language can be mathematically circumscribed, and ultimately proscribed. Language will be mechanically generated and circulated, and humans will be mechanistically affected by it without having the deluded notion of being able to change it or express themselves through it. Rather, language will articulate the capabilities of humans; it will program them like any other machine. In other words, language is not a medium for collective practice or the world spirit or moral freedom or the articulation of new concepts or the spontaneity of the human mind; it is instead an inefficient mask sitting over a complete and fully determined world, in which spontaneity and spirit are illusions.

If language can be completely reified as math, then it can no longer express new possibilities; this in principle would seem to make LLMs attractive to authoritarians and their aspirations to permanent rule. If the development of LLMs eventually lives up to tech-company hype, language would remain useful for issuing commands and documenting what exists but not for dissent or for articulating negation, for imagining something other than what appears to be. The very existence of LLMs could be used as a cudgel to dissuade people from using language or writing as a means of thinking, of free play, even when their output itself is not being deployed to clog up all the accessible media channels with artifacts of the status quo. The models would “work” not to the degree that they synthesize accurate or plausible responses to queries (this won’t matter because meaning wouldn’t really matter), but instead to the degree that they can be evoked and deployed to discourage people from using thinking freely in language, from sustaining language as an instrument of free thought.

In 2018, futurist Yuval Harari posited that AI would facilitate “digital dictatorships,” but his argument is based not so much on the impoverishment of language and thought but on the idea that surveillance and voter manipulation could be automated and information control could be centralized without emptying that information of its significance. “The main handicap of authoritarian regimes in the 20th century—the desire to concentrate all information and power in one place—may become their decisive advantage in the 21st century,” he claimed. At that point, decisions would be taken out of the hands of citizens, who would be made to be fully dependent on the all-knowing algorithmic autocrat. “Human life will cease to be a drama of decision making, and our conception of life will need to change,” he argued. “Democratic elections and free markets might cease to make sense. So might most religions and works of art.” (As if “free” markets currently “make sense.”)

Harari presumes that the aims of AI developers are indeed plausible, that data is unproblematic to collect, that language really can be solved and reduced to a kind of fixed programming — that adding more and more data to a single model will eventually yield a complete picture of reality, understood as an already fully determined totality. Thus from his implicit viewpoint, it won’t be the exercise of authoritarian violence that backs the imposition of LLMs, but their intrinsic validity in and of themselves that will make them irresistible to us, making the emergence of an authoritarian regime capable of administering an AI-governed world inevitable.

That is, he appears to take for granted that LLMs will succeed in mathematizing language — language is not a living thing but a pile of stones whose true nature and distribution will eventually be completely calculated. People — at least, “ordinary people” whose “natural stupidity” will have become “empowered” by AI — will adopt predictive models as their decision maker not because they are forced to by humans with coercive power over them but because they will recognize that AIs are wiser than they are: '“We might willingly give up more and more authority over our lives because we will learn from experience to trust the algorithms more than our own feelings, eventually losing our ability to make many decisions for ourselves,” Harari writes.

From Harari’s point of view, AI invariably makes data collection more efficient and gives more information and power to the centralized entities who reap its harvest. But as political scientist Henry Farrell details in this rebuttal, drawing on James Scott’s Seeing Like a State, there is no reason to assume that “the strengths of ubiquitous data gathering and analysis” will automatically reinforce “the strengths of authoritarian repression to create an unstoppable juggernaut of nearly perfectly efficient oppression.” Instead, the efforts to make ordinary people more legible will make them more opaque, and the gaps in surveillance will become more salient and significant. More data will not lead automatically to a clearer picture capable of “solving” reality, but will instead lead to more errors and oversimplifications in how that data is processed, even if you presume that data itself can somehow be taken as an objective picture of the world rather than something distorted by the means and contexts of its collection.

Machine learning, Farrell argues,

can serve as a magnifier for already existing biases in the data. The patterns that it identifies may be the product of the problematic data that goes in, which is (to the extent that it is accurate) often the product of biased social processes. When this data is then used to make decisions that may plausibly reinforce those processes (by singling e.g. particular groups that are regarded as problematic out for particular police attention, leading them to be more liable to be arrested and so on), the bias may feed upon itself.

This is the “garbage in, garbage out” problem, and at a certain point in the pursuit of total information awareness, all data is garbage.

Rather than strengthening “digital dictatorships,” then, reliance on machine learning, Farrell argues, would render authoritarian regimes both more unstable (they would have an inaccurate understanding of conditions, including the intensity of discontent) and more cruel (they would be dependent on increasingly biased data).

A recent paper on “The Digital Dictator’s Dilemma” by Eddie Yang (described by Farrell here, and by Cory Doctorow here) aims to substantiate that general idea by looking at censorship regimes in China. Yang emphasizes the possibility that “citizens’ strategic behavior in the face of repression diminishes the amount of useful information in the data for training AI.” So if repression took the form of imposing algorithmic decision making and generative models on populations, it would not necessarily drive those ordinary people to embrace their natural stupidity but to learn new forms of resistance that can shift the ground beneath those models. (This dilemma often comes up in content-moderation debates, when some form of machine learning is proposed as a solution, as if objectionable content were some stable, fixed category, and human ingenuity were incapable of finding ever new ways to be offensive.)

This suggests that the stakes are high for ideologizing AI — a lot depends on whether the imposition of generative AI is experienced as repressive or as convenient. If it seems convenient, data will presumably be less garbage-like and more useful for the purposes of social control. If it seems repressive, it will create a population intent on making more garbage. So the more that we can be convinced that AI is potentially useful and labor-saving, the more effective machine learning and surveillance will be at administrating an authoritarian society. And it may be that there is no middle ground — there are no “convenient” uses for generative models that don’t also augment its repressive possibilities. (Of course, I would argue that convenience simply is a form of repression, the sunny face of deskilling and disempowerment.)

Another implication is that data is more useful to models and autocrats when they refer to unthinking people — when people are understood to behave like things rather than living beings capable of free choices. Part of the ideology of generative AI is that people are fully determinable through a set of behavioristic laws: They don’t have (i.e. shouldn’t have) the freedom to alter the meanings and interrelations of things that have been captured in a model; they can’t use language in novel ways that upset the parameters of an LLM; they can’t exist in any ambiguous or ambivalent states. All the data pertaining to people is one-dimensional and precludes their ability to produce it tactically, with obscure intentions. In fact, the logic of AI presumes that humans don’t really have intentionality at all, that freedom is an illusion. Generative models are built on the idea that representations of the world don’t at the same time change that world. But that is only true if models are capable of abolishing subjectivity.

Farrell points out that “when the world begins to fill up with garbage and noise, probably approximately correct knowledge becomes the scarce and valuable resource. And this probably approximately correct knowledge is largely the product of human beings, working together under specific kinds of institutional configuration (think: science, or, more messily, some kinds of democracy).” Or, language itself. Generative models are “parasitic” on those configurations while purporting to replace them and render them moribund. The ultimate aim of an LLM is to make human beings willing to do without language altogether.