Ex noumena

In a recent piece for the New Yorker about Unreal Engine, which is used to render the graphics in video games and virtual sets for film productions, Anna Wiener paraphrases film studies professor Julie Turnock, who makes the point

that visual realism isn’t always about imitating “what the eye sees in real life.” She brought up filmmakers’ use of visual elements like shaky camerawork and shafts of light glittering with dust motes—gestures toward realism whose presence is sometimes gratuitous, or even defies logic. (These have also become common in video games: there are no cameras in video games, but there are lens flares.) Turnock traces this back to efforts in the seventies and eighties to make the early “Star Wars” films look gritty and naturalistic. “There’s a whole series of norms that have grown up around what makes things look realistic,” she said. Some attempts at realism, it struck me, were so realistic that they could only be fake.

This is fundamental to all forms of realism, no matter the medium. Representations can be ruined by being “too real.” An unfaked video of an actual tornado — like this one, which Nathan Jurgenson forwarded to me — can seem too realistic to be real. He wondered if such clips might be thought of as “deep reals,” what happens when documentary reality has the grammar of fakeness, of CGI simulation.

How real a piece of media seems to be is inseparable from the intentions an audience imputes to whoever made it and whoever distributes it. The point of a “deep real” would be to frame actual events in a way that would help them circulate — that is, to subordinate events to their potential status as content. When anything goes viral, it makes sense to doubt its veracity because of the manipulated nature of algorithmic feeds, where everything feels staged and calculated for likes. In that context, the instability of “realness” as a foundation for interpreting media becomes more pronounced. The barometer for relevance, the deciding factor in whether you see something, has little to do with “reality” as such. Instead, algorithmic feeds tend to increase the tension between events and content, between what holds our attention and what actually happened somewhere.

Realism is that tension. Realistic works don’t just document everything; they frame events (fictitious or not) to seem like documents, which an audience can then enjoy on those terms, on the level of “fidelity” or “authenticity.” To experience something as “realistic” depends on the governing conventions, the “norms” (and not some set degree of empirical facticity). These norms change much the way fashion changes as certain ways of signifying fashionability get worn out. Authenticity marketing deliberately conflates fashion and realism, capitalizing on the idea that one is compelled to keep up with changes in what “feels real,” even in relation to oneself. Implicit in every commercial reference to “authenticity” is a rejection of the real as residual, as what’s just there, independent of people’s efforts to frame it or present it. To “be real” or to “be oneself” means accommodating the norms of realism as they play out in different trends regarding what products are “authentic” and what kind of relationships or practices are “genuine” and what forms of self-presentation are “sincere” and so on.

You can’t just come to terms with yourself once and for all. What is “real” must continually and deliberately be reproduced, according to standards that are always modulating in direct response to that effort. The self is intrinsically real, but its realism must be deliberately achieved.

Despite the ever-shifting social conventions of realism, the effort to produce “more realistic” media is often treated as a purely technical challenge, a matter of getting more definition, more resolution, more pixels, more polygons, more detail. There is an ideology in this version of “realism” that promises an escape from the endless effort required of us to reassert the reality of our perspective. Instead, there will be perfect simulations, perfect mediations, that make individual perspectives superfluous. From within this ideology, to be “realistic” is to achieve a perfect match between what is recorded or simulated and its counterpart in reality, capturing it exactly and completely, as if that were self-evidently possible — as if the real were a quantity, as if there were some less than infinite amount of substance to any given moment.

Hence the tech executives Wiener profiles in the piece toss off claims like this one, from Epic Games’s Tim Sweeney: “The solution to fluid dynamics and to fire and to all these other phenomena we see in the real world is just brute-force math. If we have enough computing power to throw at the equations, we can solve them.” Reality is a quantity, and quantities can be fully expressed in formulas. Life can be calculable, and that is what people want to see in “realistic” entertainment.

Sweeney also laments that “we don’t even know the equations we need to solve in order to simulate humans,” and hopes to discover them by deploying more scanning equipment on human bodies.

To create one model, Epic’s researchers gave an actor a full-body MRI, to scan his bones and muscles, then put him on a stage surrounded by several hundred cameras to capture the enveloping tissue. To simulate his facial movements, the researchers put sensors on the actor’s tongue and teeth, placed his head in an electromagnetic field, and collected data on the ways his mouth moved while he talked.

To simulate humans, you don’t need to understand how they think or why they move, you just need to invent equations that can account for how some specimens have evidenced movement in the past. Simulating a human, that is, is no different from simulating a basketball.

Once all the data is in, AI people (“digital twins”) can be properly put to work, as this unhinged Wall Street Journal report anticipates: “Enterprises are starting to invest in the idea of using AI to digitize and monetize some aspects of humanness.” I guess humanism was good for something after all.

The logic behind those efforts is that humans and their environments can be so thoroughly and accurately simulated that you can draw conclusions about what real people will do in real situations from how equations play out when fed different kinds of data. They are predictive, generative models for “humanness” abstracted away from the individual dignity of any particular human, thus the “AI people” can be aggressively reduced to “information about their brand preferences and shopping habits.” If enough AI people are deployed as mirrors of ourselves, we may model our intuition of what is “realistic” on them and come to recognize that we ourselves are no more than brand preferences and shopping habits.

To believe that this general approach to simulations makes them “more real,” one must not only believe that a person is no more than a finite sum of data points. One must also assume that human perception is entirely passive, untempered by consciousness, by imagination, by reason, or by any other cognitive faculty — that there is just raw sense data that passes through the perceiving organs and renders their exact content in some staging “volume” of the brain. (Though where one might posit such a space is hard to figure out. It’s not clear how Epic Games would be able to install its lidar sensors and its batteries of cameras there to capture it for simulation.) Reality is a set of quantities and co-ordinates that can be transmitted from entity to entity and medium to medium without any gaps; there is no distinction between subject and object, no need to accommodate different perspectives. The idea that there is any rhetorical dimension to what is experienced as real — that what seems “authentic” derives from persuasive details, not accurate ones — is utterly rejected.

Similarly, this understanding of what’s “real” confuses concepts with empirical things in themselves. Wiener details how the Epic subsidiary Quixel travels the world “scanning in” what it takes to be representative examples of things and turning them into digital “assets” that can be deployed as needed into rendered environments on screens or into Shen Yun–like “volumes” to provide dynamic backdrops for entertainment production. “The ideal assets are iconic, but not distinctive: in theory, any one of them can be repeated, like a rubber stamp, such that a single redwood could compose an entire forest,” Wiener explains. Scanning purports to be captured empirical reality as it is, but what it is doing is reifying human concepts of things in specific ways, as certain configurations of data to be fed into certain sequences of equations. The guy scanning the redwood is turning that specific tree into “treeness.” The full-body MRI extracts “humanness” from the human.

What simulation engines like Unreal offer, then, is the experience of navigating a fully pre-conceptualized world, where not only is the movement of everything depicted in space predetermined by a complete set of equations that account for every dimension and aspect affecting the material world, but also where everything has already been synthesized and schematized into its specific meaning. As Nathan put it, “the trees are very tree, the road is very road, everything pops — an environment made of Platonic forms that almost evokes a nostalgia for our own internal expectations of the world that are never fully realized away from the screen.”

That nostalgia is the feeling of realism. But simulators don’t produce it through accuracy to documentary reality; they do it by presenting concepts as though they were empirical realities. In an immersive simulation, you can navigate a world of concepts as facts, as though they were capable of being simply perceived rather than having to be thought. The nostalgia is a by-product, possibly an unintentional one, of trying to save people the effort of having to understand anything.

If for Kant, the capability to form schema is “an art concealed in the depths of the human soul, whose real modes of activity nature is hardly likely ever to allow us to discover,” for Epic Games it is just a matter of more cameras and more 3-D scans. Just keep taking pictures of the most cloud-like cloud you find until those “modes of activity” become superfluous along with the notion of the “human soul.”

One might optimistically think of this as escaping from Plato’s cave to see not the shadows but the true, ideal forms of things (as arbitrated by tech executives like Tim Sweeney, who, as a co-worker suggests to Wiener, may have a “God complex”). Or one might see this as being denied the opportunity of exercising one’s own imagination, one’s own subjectivity. In Critique of Judgment, Kant develops his argument from the first critique that there are “no rules for judgment” — no way to simulate the process of “experiencing” things before the fact — and more or less equates the ability to tentatively conceptualize things in a state of “free play” with the possibility of aesthetic experience, of pleasure in the endless effort of balancing sensation with comprehension. Human freedom boils down to that irreducible gap between things and our concepts of them, and how it is possible to find oneself suspended in that gap where concepts are invented and expanded and interwoven.

Enjoying “realism” is an aesthetic experience in this sense — it is not a moment for validation and authentication, but a suspension between the real and the possible, and ideal and the contingent. It is a feeling of having our imaginative capacity being held open by something that resists but also invites total conceptualization. The pleasure it offers comes not from perfect identity with “things in themselves” but in how a representation compels this kind of “free play,” a suspension of disbelief, an awareness of the work and the stakes of “synthesizing the manifold” in particular ways.

Realism, when it’s effective, doesn’t preconceptualize the world so much deconceptualize it, so that it’s seen anew. Unreal Engine’s would have us accept that there are no judgments, only rules, and that tech companies are hiring all the world’s “engineering talent” to carve them into silicon. Their monopoly of “compute” will make them the only entities that can calculate the real. As this Bluesky post puts it, “AI companies have commodified the shadows on the cave wall and are selling them back to the cave's inhabitants as a device that lets them behold the true world.”

But Unreal Engine succeeds in producing “realism” only because it fails to perfectly duplicate the real, which would only make its media vanish altogether. No representation of reality can be total, because it would cease to register as a representation and “realism” would just be reality. But “realism,” a heightened sense of the significance of things seeming how they are, is often preferable to reality, in which the mundanity and entropy of things are inescapable and inexorable. (Welcome to the desert of the real, etc.)

Sweeney seems to want to collapse the distinction between map and territory: “We could have a 3-D map of the entire world, with a relatively high degree of fidelity. You could go anywhere in it and see a mix of the virtual world and the real world and any combination of real and simulated scenarios you want there.” But this would seem to invite the mise en abyme problem, where there is a map of the map of the map to infinity, or the doppelgänger problem, where infinite clones of objects drown the world, rendering conceptuality, if not thought itself, impossible.

While one of Epic’s engineers asserts that “the holy grail of all this is to cross the uncanny valley,” that seems entirely backward, if the goal is to give people pleasure. The aim instead is to perfect the uncanny valley, to make its uncanniness resonate in precisely the right way to evoke the same experience we have whenever we become aware of how we are trying and failing to exhaustively conceptualize what we are sensing.

It is reassuring to be able to put aside the map and trust that the territory still exceeds it — that there is still something real that goes beyond what is fixed and determined. This allows us to believe we can still exit mediation and simulation and be “in the real world.” Despite itself, Unreal Engine re-enchants the unknowability of reality. The more detailed it gets, the more it misses its mark.