I’m ashamed to say that I was largely unaware of semiotic language formation for the bulk of my life – I didn't even hear the term semiotics until I read Gibson’s Patten Recognition at the age of 21, hung over, sleep deprived, and jet lagged, a fine way to read that particular book. Even after I’d been introduced to the term it remained buried somewhere in my subconscious for the most part, never even approaching the forefront of my thought process until I began teaching composition. In composition, which focuses predominantly on conventional “verbally literate” texts, that is to say texts made up primarily of words, concerns about visual literacy (which normally deals with texts composed primarily of pictures) are often given short shrift, if they’re engaged with at all. When you think about it, that's pretty insane: most people are far more adept at symbolic or semiotic visual literacy than they are at verbal-visual literacy. After all, semiotic literacy engages directly with visual memory through pattern recognition, without using any of that messy locative memory shortcutting that our jury-rigged ape brains use to generate visual-verbal literacy equivalencies. In a sense, that makes it much more challenging to actually engage with semiotic and symbolic language: we’re used to just accepting the language of semiotics in the world around us unconsciously, absorbing information without realizing it. Reading is a conscious act, a challenge for many people, and fixating on aspects of a process you’re already aware of, a process you’re forced to consciously work on improving during some phase of your educational development regardless of who you are, can be much easier than forcing yourself to analyze the nuances of a process you engage in almost as reflexively as breathing. But when you discuss semiotic or visual literacy in concrete terms, students who had trouble with general literacy begin to acquire deeper critical thinking skills they can utilize to, in turn, improve weaker skills that they lacked the capacity to self-assess their progress in previously. It’s far from a perfect system, and it’s challenging: the study of semiotics is never really engaged in in a thorough critical framework, and those who do study it often do so for the sake of manipulating its frames through the generation of branded material in fields like design and advertising, rather than the generation of discourse surrounding the response of individuals to that brand. It's an unfortunate state of affairs, because breaking down the function of semiotic response mechanisms can be extremely edifying. By making the invisible visible to an audience, you’re doing more than just deconstructing symbols: you’re deconstructing the response-feedback loop surrounding those symbols.
So what the fuck does any of this have to do with video games?
More than you’d think. Video games are essentially engines for instruction. They all but categorically teach players to apply patterned solutions to problems, often stacking mechanics on top of one another through sustained play that prompt increasingly complex solutions in response, generating a feedback patterns promoting general critical thinking and problem solving skills. Think of Half-Life 2’s helicopter battles. First you’re on a boat. You’ve got no rockets. You’ve got no choice, really, but to run the fuck away from that god damn helicopter. Alyx is there giving you tips. She’s acting as the voice of the developers as they teach you how to avoid being straight up murdered by that god damn flying death machine. After that, you’ll be given a handful of rockets and asked to fight another helicopter. Here you learn how to aim and fire while evading, utilizing cover to keep yourself safe and firing at the helicopter when opportune moments presents themselves. There’s a steady mix of aiming and dodging and timing there, but the basics are coming together for you by now: you’re figuring out how to kill these helicopters, and how badly you can fuck up without thoroughly screwing yourself. Eventually, you’re forced to fight these helicopters in different scenarios: in some, you’ll be in mostly open ground, and your timing as you move from piece of cover to piece of cover is key. Sometimes, you won’t have access to any rockets or anti-aircraft weapons at all, and you’ll be forced to use the grav-gun to fire mines back at the helicopter as it drops them. Sometimes you’ll fight weird fishy-helicopters that don’t drop mines, and you'll be forced to use the weapons at your disposal and objects scavenged from your environment. Eventually, you’ll fight them with nothing but your souped-up grav gun, at which point you’ll only have one weapon to use against them: energy orbs, which you’ll have been trained in the application of during a series of exhausting earlier battles.
Half-Life 2 repeats this pattern a number of times across a variety of mechanics, but the core pattern of challenge-solution-expansion is uniform across these examples. Most games use this pattern to teach players how to interact with their mechanics, though not all of them are as deliberate or thorough with their application of this strategy as Half-Life 2. Nor are the anywhere near as transparent, thanks to Half-Life 2’s seminal and still-enriching “commentary tracks” scattered throughout the game. Valve took the time not just to craft these arenas, but to tell us that they were internally called “arenas,” and that they were oriented around giving players a particular object lesson, especially in the first half of the game. They made what McGonigal and Gee discussed in broader terms very specific: they demonstrated the instructive capacity of games in a directive, measurable way. They've even got outcomes metrics, something that Gee lacks entirely and that McGonigal, who I love dearly, is somewhat fuzzy on when she brings them to bear in her talks.
Video games are also highly reliant on visual languageLet’s look at Half-Life 2 again, with its broad scope of art design. Terrain, construction, and environmental symbolism in Half-Life 2 presents players with a bevy of useful information. Video games rely predominantly on visual literacy skills, often to the point that writing becomes a secondary or tertiary concern. Much of the information we’re given about our world emerges from snippets of visual language: a vortegaunt standing by a trash can fire, an aging priest wearing a pair of Chuck Jones originals as he blasts away headcrab zombies with a shotgun, a horde of zombines pouring out of a troop transport, surging towards you with grenades in hand. These bits of visual data all operate on a semiotic layer, either drawing on existing symbolic or imagistic information from the real world, instructing players in the meaning of visual frameworks within the game, or building on that understanding of said visual frameworks to convey important information about the world players are moving through. Even the laser sight trail of a sniper rifle is a semiotic clue, providing players with information about where, when, and how they can move through a space. When you’re given a chance, in Episode 1, to use those sniper rifles to your advantage through Alyx, the semiotic message is clear: the combine are on the ropes, and their tools have become yours. The symbols in the world that represented dangerous obstacles now represent handy resources that can get you out of a potential jam. The symbol, and its inversion, achieve a narrative process more effectively than a cutscene or some dialogue might.
Half-Life 2 employs semiotic information to give players instruction and to tell stories in the world around them, sometimes using subtextual information contained in a character’s style of dress or the condition of their equipment to give us background or history about a given individual. Portal takes it even further, creating an environment rich with semiotic language, the corruption of which, as the plot continues, informs players about both the state of their world and the state of GlaDOS’ authority. As players begin to strain against it, the veneer begins to peel away, revealing information about previous occupants of GlaDOS’ experiments, as well as information about the world you’re in (through the delightful frame of an abandoned power point presentation, one of the more concrete indicators of slapdash half considered perspective in the modern world.
All of these references, created by artists and writers, inform these worlds. They’re a huge contributor to the kind of world building games are invested in, and a big part of the particular literacy that games encourage in their participants. The same way reading a particular kind of book might encourage students to look for certain patterns of writing to help them locate particular kinds of information, playing a particular kind of game indoctrinates players towards engaging in a particular kind of semiotic discussion with their surroundings, as well as the mechanics of the game itself. A player who frequently plays RTSes will look at the world in a different way than a player who plays FPSes, who will look at the world in a different way than someone who plays RPGs. But all of these learning structures rely on notions of subtext and symbolism that we don’t usually discuss, even when we acknowledge their presence, and the generative proto-linguistic framework that is semiotics is core to effectively discussing these subjects.
Developing an effective language to that end could be a major step towards improving games as education tools, literary and artistic structures, and generating legitimate video game criticism. I think Valve could do some remarkable things with philosophical and scientific principles given a little push in the right direction, but right now it doesn’t seem terribly interested in doing anything but generating games that the critical apparatus can’t really process appropriately. That’s also a push in the right direction: that games should exceed criticism’s grasp encourages both mediums to excel, and in the case of criticism, that’s particularly necessary.
Astute allegations of sexism and nepotism on both sides aside, #gamergate brought something to mind that I’d stopped thinking about long ago: video game criticism is broken. And while one could say it’s broken for the same reason that poetry criticism is broken (it’s almost always hinging on professional courtesy between reviewer and game-maker, and those game-makers often take especially effusive and talented reviewers on board as writers) I think there’s a more fundamental challenge facing games criticism, that of an effective critical linguistic framework. This problem has been around for a long time, ever since gaming magazines tried to review games with such categories as “fun factor” in an attempt to represent their value as experiences, and it’s sustained in the manner in which the apparatus continues to rely on numeric or graded scores for artistic products, an absolutely absurd conceit when you consider that the same process is being applied to Call of Duty and Depression Quest. There have been pushes away from this, often in the form of heated issue pieces, like Leigh Alexander’s critiques of the representation of women in video games, and Stephen Totilo’s highly conventional measured journalistic writings which aim to treat games with the same level of discourse as any other artistic medium, but the bulk of the industry seems content to avoid any kind of real critical discussion of games as an art form in favor of utilizing a set of in-house comparisons and slapdash subjective linguistic frameworks that address outcomes, rather than processes by which games achieve their goals. Even people like Anita Sarkeesian, whose remarkable videos have polarized elements of the enthusiast community, rely on linguistic frameworks that struggle to convey their own points (though, to be fair to Sarkeesian, a great deal of that can be attributed to the massive scope of her undertaking, and the breadth of her audience).
When I talk about developing a lexicon for discussing semiotics in games, I don’t just want to see it applied there: the discussion could easily escape the medium, and take root in any number of semiotic dominated fields. Discussing the impact and evolution of branding, discussing the way that mechanical engineering and design shift over time, discussing the manner in which signage functions in different cultures; these are all valid venues for the application of semiotic discourse, but they’re far more often discussed in terms of their outputs (and how those outputs might be gamed or manipulated) than their moving parts. I don’t know how a critical or academic language for semiotics could be generated, short of a massive undertaking, but when I consider the potential impact, it seems worthwhile. Games have long stood as a tremendously influential medium and a remarkable instructive framework; participatory frameworks are almost always super effective. But without an effective language to outline the flow and function of semiotic language, we can barely discuss things like Sean Tejarchi’s LiarTown USA with more depth than nodding and saying “that’s funny.” Even Jezbel.com, where feminism goes to die, managed to miss a joke at its own expense, recommending that people buy a “Social Justice Kittens” poster that lampoons the very kind of language and infantile discursive discourse that Jezbel relies upon. That says something about the state of semiotic discourse: we’re capable of engaging with the modes and models it presents, but when it comes to reflecting on the function of those meanings, or even just deconstructing them in the broadest senses, we seem to consistently come up short. It’s just especially apparent in games, and until we have a language for discussing the frameworks games rely upon for generating their discourse, we’ll always be discussing it in childishly broad terms, struggling to express the profundity of what our experiences have shown us as consumers of this, the most nascent of artistic mediums.