Augmented Environments and New Media Forms

Ubiquitous Media

Media are everywhere. Other advances—particularly in biotechnology—may ultimately have greater social impact and promise to remake what it is to be human. But if we look at the most visible changes that our culture has undergone in the past fifty years, they are in the realm of media. Many, perhaps most, readers of this article will remember a time when there were no smartphones or tablets; when cellphones were bulky and generally owned by businesspeople and so-called early adopters; when it was impossible to watch a television series or film on your computer; when LCD or LED flat-screen televisions were rare and expensive. Today people in the advanced economies carry media devices everywhere they go. Bars and airports have television monitors almost literally in every direction that you look. Elevators in office buildings have small screens to watch cable news and advertising as you ascend. Even some gasoline pumps now offer video entertainment (and of course advertising) as you fill up your car. Meanwhile, as for 2015, three digital media giants (Google known as Alphabet, Apple, Microsoft) led all other companies in market capitalization; Berkshire Hathaway Inc, a financial company, and Exxon Mobil came in fourth and fifth (Price Waterhouse Cooper, 2016). The sixth company, Facebook, whose social networking site is nothing but virtual entertainment, continues to climb toward an active user community consisting of one-quarter of the population of the planet. These are remarkable indications of the economic as well as cultural power of contemporary media.

For Hollywood and the science-fiction community, however, this is only the beginning of a media-saturated future. In their sometimes utopian, sometimes dystopian imagination, media will eventually augment every facet of our lived world. The film Minority Report (2002) described such a world for us. As the main character, John Anderton, walks through the corridors of a mall, his retina is scanned for identification. The walls and space around him then come alive with projected advertising tailored to his presumed shopping preferences; mixed reality products and services, Sirens of the marketplace, call out to him by name. Google, Facebook, and many other sites already tailor advertising, but of course only on our computer screens. And the Microsoft HoloLens, as well as the VR headsets such as the Oculus Rift and the HTC Vive, already have the capacity to surround us with 3D images. A film such as Minority Report might lead us to wonder whether we all face John Anderton’s future, greeted by digital advertising wherever we go… Is this the next step?

As early as 1991 in an article appropriately entitled “The Computer for the 21st Century,” Mark Weiser at Xerox PARC proposed the term “ubiquitous computing” to describe the way that the networked computers would embed themselves in our physical, economic, and social fabric. He envisioned a world in which our appliances, cars, and buildings would be equipped with sensors and networked together. They would communicate not only with us but also directly and invisibly with each other, anticipating and meeting our needs. Twenty-five years later, Weiser’s vision is being fulfilled as the so-called Internet of Things: that is, devices scattered throughout our environment gather data on us as well as for us. The reigning business model for the Internet of Things is to learn as much as possible about the consumer in order to provide services and products that the consumer will view as valuable. In the manufacturing sector the Internet of Things includes all sorts of sensors and controllers for industrial processes. For consumers the smart home is gradually gaining in appeal: smart thermostats and lights, motion sensors for security, doors that can be locked or unlocked remotely from the Internet, and so on, are increasingly popular ways to augment our houses and apartments. But by far the most popular consumer services and products are new forms of media. Our phones, tablets, smartwatches and other wearables, and laptop computers are the essence of ubiquitous computing today, as we employ them to refashion our media culture. And Weiser was wrong in one important respect: ubiquitous computing devices do not need to embed themselves silently in our world to count as ubiquitous. As Steve Jobs realized (and this is what made Apple the leading company on Forbes’s list), consumers do not want their media devices to disappear: they want beautiful, sleek mobile phones that make a personal fashion statement as well as delivering excellent results. Furthermore, we should not think of ubiquitous media as including only the latest digital products. Today’s media culture still includes many “traditional” forms: print books, newspapers, magazines, photography, film, radio, and television—all with digital counterparts. The interplay of all these forms constitutes ubiquitous media today. Ubiquity means not only that media forms are everywhere in our daily lives, but also that they combine and interact with each other in every conceivable way.

Just as Mark Weiser’s notion of ubiquity helps us to understand contemporary media culture, we can also look to an earlier student of technology, Marshall McLuhan. For all his excesses and inconsistencies, McLuhan in Understanding Media (1964) provided us with a key insight when he observed that new media extend our senses and alter their ratios. This is not to say that we accept McLuhan’s brand of what is called “technological determinism.” We should not have to think of radical media technologies (such as the printed book in the fifteenth century, the film camera in the twentieth, or the smartphone now) as autonomous agents beyond our control that alter what it means to be human. A more productive way to understand the process is to acknowledge a reciprocal relationship between the characteristics of each new media technology and the way we as a culture deploy that technology.

Both as inventors and as consumers, we make choices about what constitutes the new medium and how we will use it. For example, we decided as a society that radio, which was originally conceived as a two-way medium for individual communication (what we call “amateur” or “ham” today) became (in the United States from the 1920s on) a one-way medium for commercial broadcasting of music, series, news, and so on—extending our ability to hear but not to talk back. We decided as a society that film, which flourished in its “silent” form, needed to become an audiovisual medium in the late 1920s and later in the 1950s to adopt anamorphic, wide-screen projection.

Currently we are deciding how our ubiquitous media will augment our sensorium and expand forms of entertainment and communication. We are configuring mobile phones, wearables, and mobile and remote sensors into platforms that address our senses of sight, hearing, proprioception or embodiment, and touch. As discussed below, ubiquitous media today often combine many of our senses and human capabilities, and we use our devices simultaneously in ways that put multiple senses in play. It is also important to remember that the ubiquity of media does not mean that any one medium replaces all the others. Although videogames and social media are among the most popular forms today, there are still vast communities of readers, filmgoers, and television watchers. Our media culture today is not only ubiquitous but also heterogeneous, and we need to appreciate how the contemporary versions of so-called traditional media interact with digital media.

Remediation

In the late 1990s, Jay Bolter and Richard Grusin (1999) introduced the concept of remediation to explain how media affect each other in our media economy. Even then it was apparent that the digital media of the time (such as the Internet and videogames) were in a constant dialogue with traditional media (especially film and television). The dialogue was sometimes cooperative, sometimes competitive. Producers of new digital forms borrowed formats, styles, and genres from the earlier established media forms, while at the same time claiming that the digital offered something new and exciting. For example, to advertise their products business websites borrowed the techniques of graphic design that consumers had been seeing for decades in magazines. At the same time business sites pioneered online sales, which combined the characteristics of mail-order and television shopping, but offered a new sense of immediate response. News websites came online in the 1990s, sometimes owned by and explicitly remediating the printed newspaper (such as The New York Times). These websites drew on the reputation and reproduced the written articles of printed papers. But they promised a new level of timeliness. A printed paper came out once or a few times a day, whereas new websites could be updated as the news broke.

The process of remediation has continued in the past two decades. In fact it is a defining feature of our media economy. As new forms are introduced, they are explained in terms of existing media—most often through the formula: this new form is like one or more older ones, only better. In 2007, in one of his famous product roll-out presentations, Steve Jobs introduced the iPhone by saying that today Apple was introducing three new products: an iPod with touch controls, a revolutionary new mobile phone, and an Internet browser (YouTube, 2007). Teasing the audience, he showed a graphic of a monstrous hybrid: an ungainly rectangular slab housing a small screen and an old-fashioned rotary dial for the phone. The next slide presented the actual first iPhone in which these features were absorbed into a sleek, modernist design. The transition between those two images was a perfect emblem for remediation, showing how Apple’s new iPhone absorbed, refashioned, and ostensibly improved upon the media it was meant to replace. The iPhone was like a phone, only better; like a music player, only better.

In the same presentation, Jobs also claimed repeatedly that the iPhone was revolutionary, a device that would “change everything.” Digital media writers and digital companies often adopt this rhetoric. According to them, it is not only the iPhone, Facebook, or Google that have changed the world we live in; each new smartphone with a better camera or fingerprint or retinal recognition, each new interface on Facebook is called revolutionary. And conversely, if a new release of the iPhone is not revolutionary, but is merely an upgrade, then the digital world is disappointed. The rhetoric of revolution is a holdover from the middle of the twentieth century, the era of high modernism, which was defined both in art and popular culture by a cult of the new. The assumption was that in order to be significant at all, each new work of art, each new invention had to be unique, utterly original, not just an improvement, but a breakthrough. The modernist assumption is still widespread, despite the fact that our media culture is now widely characterized by different creative impulses—by remix and remediation, by borrowing and refashioning earlier media and media forms.

Even the newest of new digital media forms are remediations, borrowing and refashioning older forms. The most successful of these innovative forms today are videogames and social media, and both have clear antecedents in analog media. Videogames are obviously the new versions of an entertainment form, the competitive game, that is thousands of years old. Some videogames remediate board games; others remediate puzzles; still others arcade shooting and skill games. Some genres of videogames, such as the role-playing and adventures games, also borrow both storytelling and camera techniques from film. And remediation works in the other direction as well: that is, from digital media back to traditional media. Film can remediate videogames, when, for example, movies are made based on videogame franchises such as Tomb Raider or Resident Evil. Social media are now the most widely used digital form with billions of users, and they are also among the most innovative. And yet each of the major social media applications has roots in earlier media traditions. Facebook began as a remediation of the (Harvard) college yearbook, with its format of picture and textual information about students. Image-sharing sites, such as Instagram and Flickr, remediate the family photo album and photographic journals. Twitter combines the terse quality of the sms with the broadcasting feature of mass media. All of these forms are recombinations of earlier print or visual media as well as earlier electronic ones. All of them redefine or confuse the spheres of privacy and publication from these earlier media. And all of them add the particular digital characteristics of interactive and (almost) immediate communication.

After decades of false starts, Virtual Reality has recently become a widespread media form with the introduction of inexpensive headsets such as the Oculus Rift and the HTC Vive as well as Google Cardboard and software that converts smartphones into makeshift headsets. These devices are now being offered as ideal platforms for remediated videogames such as first-person shooters that were originally played on the screens of desktop computers. There is also growing interest in VR movies, such as those created by filmmaker Chris Milk and others (VRSE.com). These VR movies are remediations of traditional film genres such as the documentary and the horror film. It is the claim of VR filmmakers that its 360-degree video puts you “in the scene” as never before. This form is supposed to give you a greater sense of immersion and therefore emotional involvement in the action. This is exactly the point that Chris Milk makes in his TED talk (2015). The claim of greater authenticity, reality, or engagement is a typical remediating claim made by creators of a new medium. VR movies are like traditional film, only better, their proponents argue, because they heighten your empathy. In one VR movie called The Displaced, for example, child refugees from conflicts in Syria, South Sudan, and Ukraine are shown in their environments; we as viewers experience the refugees’ world in 360 degrees.

The introduction of new media is always a process of isolating elements of traditional media and reconfiguring them in new ways. The new configuration emphasizes qualities that may have been latent in the earlier media, but it also shows us where those earlier media fell short. The new media form seeks to make up for their shortcomings and provide a new experience that goes beyond what was available before. Thus, the next steps in media culture, the augmenting of our media experiences with mobile devices and services, will not come as a revolutionary break with the existing media, but rather as a process of cooperative and competitive adaptations of those media. We can expect new remediations to be based on existing digital forms themselves, combined and configured in new ways, although we cannot predict what those reconfigurations will be. New forms of social media are constantly appearing. In recent years Snapchat’s video and imaging-sharing service has become quite popular, initially because any images or videos that have already been accessed are erased, providing the user ostensibly with a sense of privacy and assurance that images will not be spread further. Recently, they added a kind of Etch-a-Sketch layering of various augmentations to the photos, such as Face Swap. Augmentation to already existing platforms is a common media strategy, for instance the combination of the videogame with Internet video-broadcasting has produced Twitch. We can expect these kinds of augmentations of the most popular and compelling digital forms to continue and in particular to exploit two capacities that have been dormant or underdeveloped in the dominant media of the twentieth century and that now promise augmented forms of expression and communication. Those two capacities (key augmentations) are location awareness and multisensory input (especially touch and proprioception).

Urban Computing and Multisensory Media

One way in which digital mobile media are remediating old forms is by changing the relationship between media presentation and location. Not only do mobile technologies allow us to carry media around with us everywhere we go, to experience media forms in different locations; they can also make media aware of and responsive to their location. The printed book after all has been a mobile media form for hundreds of years. You can easily carry a book with you, but the book does not change no matter where you take it. By contrast, the information in a mobile device can change because the device can keep track of its location (outdoors through GPS or indoors through Wi-Fi and other systems). Change is not necessary, of course—you can take a conventional movie with you on your tablet, watch it anywhere, and it will be the same movie. But location awareness now becomes a parameter available for the re-presentation of any media depending on the site of interaction.

Ironically, it is precisely because mobile devices allow the user freedom of movement that digital media content can now be anchored in place. One way to anchor content is through image tracking. The mobile phone’s video recognizes an image, such as a picture in a magazine or comic book, and the application can place digital content on top. This technique has been used for advertising: open the magazine to the right page, point your phone with the appropriate app at the picture, and you get to a 3D model of a Nissan sitting on top of the 2D image (YouTube, 2010). The model is animated: the doors and hatchback open and close. Marvel Comics has produced augmented comics, where pointing your phone at a particular panel delivers a new visual experience and may reveal something new in the story (YouTube, 2012). Augmented Reality (AR) tabletop games now exist thanks to image tracking: the game board is a physical graphic image, and the characters are virtual, running over the board when viewed through the phone’s screen. All of these applications are niches that may grow in the future. Children in particular may take to AR games and augmented books; media for children have long included various media elements (pop-up books, games that blur the line between reality and illusion, and so on) in order to be playful or attention-grabbing.

The second way to anchor media is through geopositioning. You can locate images, text, audio, any media content at specific locations of latitude, longitude, and altitude. A user with a smartphone can visit these locations and view the media (or hear audio through its sound system). Pokémon Go, which became a craze in the summer of 2016, is an example of a geolocated AR game. For a couple of months, at least, millions of players wandered around streets and buildings with their iPhones and Androids, looking for Pokéstops and gyms and collecting eggs, Poké balls, and potions. The game is a remediation of the age-old scavenger hunt: the appeal is that the player gets to see and pursue vivid 3D images floating in space, while the game keeps track of his or her captures.

Urban media in their digital forms are of course only the latest in a long line of remediated location-based media forms. In the 1800s exterior advertising began to appear in North American and European cities. Painted advertisements could be seen on building walls, and in the 1830s large-format posters began to be printed. These commercial advancements in turn emerged from older traditions of “writing on the world”: from the earliest signs of graffiti already in ancient Greece and Rome, if not earlier, to writing on columns (Trajan Column), buildings, and other structures to leave mediated traces. However, the modern history of urban media forms coincides with digital developments, and as mobile technology spreads and becomes increasingly available, so do the possibilities of augmenting our public and private environments. Today, technology adoption is on the rise globally: 68% of adults in rich countries report owning a smartphone; 87% use the Internet. Even in economies that do lag behind richer countries, smartphone ownership rates have skyrocketed since 2013 (Pew Research Center, 2016).

As mobile technology participates in spreading media throughout the urban landscape, the locative character of smartphones becomes particularly apparent and productive in urban settings. Applications and services can include location awareness to deliver specific and targeted information, and our location can be sent back to companies that then gather large swathes of data indicating patterns of behavior and movement. The mobile devices that we bring with us and those that are placed in objects and environments join into large networks of interconnected things and people, giving us the sense that the whole city is inhabited by media forms: some obvious and visible, others obscured and hidden. For social and practical purposes, for surveillance and control, for entertainment and commercially driven service, urban environments pulsate with digital information with which to interact.

The socio-technical condition of contemporary societies are providing communicators, designers, and programmers with a canvas for delivering content in new ways. For cultural heritage institutions, the calls for extending their offer to audiences outside the museum building have partially been answered by moving access to already digitized cultural archival material from exhibitions to relevant sites. Museums all over the world have produced quality content for mobile devices, for use inside as well as outside their buildings: for example, the Guggenheim, the Louvre, the Museum of Modern Art in New York, the Canadian Museum of Civilization, and the American Museum of Natural History. Some of these applications use location awareness to offer content for the users, providing a digitally mediated layer of information that is overlaid onto the city as a mixed reality canvas. The Museum of London’s Streetmuseum app from 2012 provides visitors with hybrid images of the present and the past as they walk through the city. The layered experience as a mediation renders the city readable, opening up institutions that many groups of citizens rarely if ever visit with a hope of democratizing content access. Other locative media experiences foreground information, communication, and entertainment. Still others focus on aesthetic explorations of mediated art in urban settings.

Early examples of digitally mediated urban art are the Los Angeles based 34 North 188 West project (Knowlton et al., 2012) and Proboscis’s Urban tapestries (2002–04), a software platform that enabled users to build connections between places and to associate digital information to those places. Today, location awareness is ubiquitous in media devices and many services and applications regularly use location as part of the data gathering aimed at providing the user with more accurate information, for navigational purposes, advertising, entertainment, game or cultural heritage experiences.

Among the next steps in ubiquitous media will be technologies that address more of our senses at the same time and mediate our environment even more intensely: for example, the HoloLens, already available in a preproduction release, with its vision of “mixed reality” (MR). A promotional video (Microsoft, 2016) shows users of the HoloLens users putting on sleek headsets to experience and interact with 3D graphics and spatial sound overlaid on their visual environment. In the video the space of the home or the office comes alive with animated “holograms”: the human body conveniently disassembles itself so that a student can more clearly see the organs, muscles, and skeletal system. Industrial designers collaborate on what appears to be a new videogame controller: it floats in front of them so that they can examine it from all directions. AR games emerge from the table or wall to provide new forms of entertainment. Video monitors follow the user as she walks around the room. The HoloLens, though, is a device that we must wear. It will only be used for limited periods of time in order to accomplish particular tasks or participate in particular games and forms of entertainment. With mobile devices that have increasing capabilities for input alongside high-end appliances and low-tech devices such as Google Cardboard, we can imagine a range of possibilities for experiences that address the whole human sensorium. McLuhan’s classic tenet that media are extending our senses as well as older media forms seems now to be more accurate than ever.

Conclusion

Science-fiction author Vernor Vinge offers us his vision of the step beyond the HoloLens in his science-fiction novel Rainbows End (2006). The year is 2025, and Augmented Reality has become the dominant technology. Those who have successfully adapted to contemporary urban life wear elaborate body suits that allow them to see, hear, feel, and interact with a world of information around them. Sensors implanted through the world, but again principally in population centers, track and respond to the humans in body suits, and all of this technology is connected to a future Internet of tremendous bandwidth. This Internet of Things and People is seamless (at least when the worldwide network is functioning properly): the graphics and registration are so good that it is impossible to tell where the physical ends and the virtual picks up.

Such a radical and thoroughgoing overhaul of our entire media environment is not going to happen in the foreseeable future. The year 2025 is less than a decade away. The problem of this prediction of a perfect technological world lies not only in the fact that technology is progressing more slowly than Vinge imagined. There is a deeper problem: no single media technology (neither Virtual nor Augmented Reality) is likely to sweep all others aside. The “next step” in our media culture will be the continued multiplication and diversification of media forms. Media plenitude will be the guiding principle, rather than one dominant medium encompassing all media practice. All the examples that we have discussed above have this in common: they are special, particularly suited for certain communities of users, certain desired uses for entertainment, work, commerce, or education. This is the way media are today: they are all tailored to particular communities, even if some of the communities are very large (e.g., Facebook). Like all science fiction and all futurology, Rainbows End is not really about the future, but is, instead a reimagining of the present. In this case Vinge’s novel is still indicative of the desire for a single media form, here AR, to take over our media world and more or less sweep away all other media. It seems likely that most of the forms that have emerged in the era of digital media—above all videogames and social media—will continue to be important parts of our media culture, although they themselves may undergo remediations for a new era of mixed reality. Mixed reality games and social media “tweets” and “Facebook pages” may begin to appear in forms that dot around the visual landscape, presented to us on a screen of something like a HoloLens. But it would not be surprising if users continue to tweet on small handheld devices as well. It would not be surprising if some segment of the public continues to read printed books, to attend movies and plays in theaters, and so on. Most, if not all, the augmentations of the past will continue, although they may have smaller user communities. There will be no single medium that absorbs all others; instead, different media will continue to flourish, each augmenting different aspects of our daily lived experience.

Select bibliography