The Critical Role of Artists in Advancing Augmented Reality

By Helen Papagiannis

Featured Article Author

Emerging technologies are not typically developed by artists, rather they are adapted for creative and artistic applications from science and engineering fields, as well as from industry. This entails issues of access, knowledge, and understanding the capacities of the technology and its constraints to exploit the technology to artistic use by envisioning novel applications and approaches, and developing new aesthetics and conventions beyond previous traditional forms.

In 1988 John Pearson wrote about the computer being appropriated by artists,offering “new means for expressing their ideas.”(1) He notes, however, that historically new technologies “were not developed by the artistic community for artistic purposes, but by science and industry to serve the pragmatic or utilitarian of society.”(2) It is then up to the artist to be a “colonizer” and to “exploit” these new technologies for their own purposes, as Roger Malina suggests, which I will later return to.(3)

https://drive.google.com/file/d/0BwnKMRXnbmm9ckozdmJLYlQ1eGc/view

Hannah Höch (1889–1978), Dada Ernst (1920-21) Collage on paper, 18.6 x 16.6 cm, The Israel Museum, Jerusalem, Israel, Vera and Arturo Schwarz Collection of Dada and Surrealist Art.

Pearson writes: “New concepts that bring new images and new materials are usually in conflict with the accepted aesthetic standard of the status quo precisely because there are few if any criteria against which they can be measured.”(4) It is the role of the artist then to act as a pioneer, pushing forward a new aesthetic that exploits the unique materials of the novel technology, in opposition to established traditions. Pearson states that computer imaging has certain “stylistic tendencies” that “have been determined by factors outside of art.”(5) However, at the time the article was written (1988), Pearson observes that artists using computers mimic art created with previous traditional art forms and “thus they fight the computer’s potential by trying to force this historical will upon it.”(6) This recalls Jay David Bolter and Richard Grusin’s concepts of remediation in refashioning a medium’s predecessor, and Marshall McLuhan’s perspective that:

Our typical response to a disrupting new technology is to recreate the old environment instead of heeding the new opportunities of the new environment. Failure to notice the new opportunities is also failure to understand the new powers. This failure leaves us in the role of automata merely.(7)

Harnessing the Specific Capabilities of AR and the Link to Early Cinema

To heed the new opportunities afforded by computer art, or any new medium for that matter, this involves, as Malina notes, “understanding the specific capabilities of the computer and creating art forms that exploit these.”(8) As an artist working with AR for over a decade, I adopt Malina’s standpoint. In order to advance AR as a new medium, it is critical to ask such questions as: “What are the specific capabilities of AR and how can they be harnessed to create new forms?” “What is unique about AR that separates it from previous media forms and, as such, what new conventions will evolve?” It is imperative that we explore and apply these unique attributes to advance the medium and “heed the opportunities of the new environment.”

I liken AR at this time to cinema in its infancy, when there were as yet no conventions. AR, like cinema when it first emerged, commenced with a focus on the technology with little consideration to content, marked as secondary. In an article comparing early film to digital cinema, John Andrew Berton Jr. identifies the critical contribution of French filmmaker Georges Méliès. Berton refers to cinema historian Lewis Jacobs crediting Méliès as the “first to exploit the medium as a means of personal expression.”(9) Berton comments on Méliès’s technical advances of the medium (fades, dissolves, and animations), and brings special note to how Méliès “did not stop with a concentration on technical achievement” but he “was busy finding ways to use his technique to carry substantial content.”(10)

Berton continues: “Even though Méliès’s work was closely involved with the state of the art, he did not let that aspect of his work rule the overall piece. He used his technique to augment his artistic sense, not to create it.”(11) Méliès maintained an artistry in the medium, being enthused by the possibilities of the technology and allowing the medium to guide his explorations and work. Méliès serves as an inspiration to my practice in AR in that the technology inspired stories and the direction of the work, but he also gave himself the freedom of experimentation and creativity to move beyond the constraints and invent new ways of applying the technology. Further, Méliès introduced new formal styles, conventions, and techniques that were specific to the medium of film; like Méliès working with a new medium, pioneer AR artists too will evolve novel styles and establish new conventions toward a language and aesthetics of AR. In working with a new technology an artist must also transcend an initial fascination with the technology, for Pearson warns of a danger artists find in “the magic of the machine” and how few seem to go beyond this “hypnotic fascination.”(12) My thoughts here turn to early cinema and film theorist Dan North’s discussion of Tom Gunning’s concept of a “cinema of attractions” where “the machine which made the pictures move was the source of fascination rather than themes and stories represented.”(13) In my artistic practice, I, too, am guilty of an initial fascination with the magic of the technology of AR; however, I believe it is important for artists to begin here, to understand and be inspired by the opportunities of the technology, then to move beyond this to advance the medium, pushing the technology and evolving new forms and content. Pearson states: “If the ideas simply revolve around demonstrating the technical virtuosity or prowess of the machine” then the artist has become a “slave” to the machine (or as McLuhan was previously quoted, as “automata merely,” failing to understand the new powers); in this case, “only technology is served.”(14)

The Importance of Working with the Technology Directly

To be able to truly transcend a fascination with the new technology and welcome conceptual evolution, the artist must be able to work with the technology directly. Early solutions to help bridge this gap between the technology and artists and nonprogrammers included DART (Designer’s Augmented Reality ToolKit) developed at the Georgia Institute of Technology. In the article, “DART: A Toolkit for Rapid Design Exploration of Augmented Reality Experiences” (2004), the authors state that: “Designers are most effective when working directly with a medium, and working through an intermediary seriously hinders (or even destroys) the creative process.”(15) The authors illustrate this point with the example of the difference between a painter directing an assistant as to where to apply paint on a canvas, rather than holding the brush oneself.

As an artist working with AR since 2005, I found it critical to my practice to be able to experiment and work with the medium directly, rather than via an intermediary such as a computer programmer (at least initially). This allowed me to better understand the limitations and possibilities of the technology, which I used as the starting point for ideas. My process has entailed first studying the traits of the technology by experimenting with the medium directly to comprehend what it does well and what it does not so well.(16) This mode of exploration guides the content development, with a story emerging that evolves from the technology, as opposed to beginning with a story that I wish to adapt to the technology. I find this approach to be critical in understanding a new medium. Once the characteristics of the technology are grasped, experimentation can push the technology in novel forms.

“Hallucinatory AR,” 2007, as a Case Study

One of my early AR artworks that adopts this approach is “Hallucinatory AR,” 2007, an artistic experiment using non-marker imagery to generate AR visuals. (It should be noted that these experiments occurred prior to any work I had done with Natural Feature Tracking (NFT); my AR work at the time was still based on marker tracking/fiducials.) The project was inspired by an accident that occurred in a previous project when the AR software was misinterpreting an image that was not an AR marker. AR content previously assigned to a marker was now appearing atop a non-marker image. This resulted in triggering unexpected and random flickering AR imagery. I was inspired to embrace this accident and explore the creative and artistic possibilities of this effect further by conducting experiments with nontraditional marker-based tracking. The process entailed a study of what types of non-marker images might generate such “hallucinations” and a search for imagery that would evoke or call upon multiple AR imagery from a single image/non-marker.

Upon multiple image searches, one image emerged which proved to be quite extraordinary. A cathedral stained-glass window was able to evoke/trigger four different AR images, the only instance, from among many other images, in which multiple AR imagery appeared. Upon close examination of the image, focusing in and out with the web camera, a face began to emerge in the black and white pattern. A fantastical image of a man was encountered. Interestingly, it was when the image was blurred into this face using the web camera that the AR hallucinatory imagery worked best, rapidly multiplying and appearing more prominently. Although numerous attempts were made with similar images, no other such instances occurred; this image appeared to be unique.

The challenge now rested in the choice of what types of imagery to curate into this hallucinatory viewing: what imagery would be best suited to this phantasmagoric and dreamlike form in AR?

My criteria for imagery/video clips were like-form and shape, in an attempt to create a collage-like set of visuals. As the sequence or duration of the imagery in Hallucinatory AR could not be predetermined because of the nature of the “glitchy” tracking of non-marker images, the goal was to identify imagery that possessed similarities, through which the possibility for visual synchronicities existed.

Themes of intrusions and chance encounters are at play in Hallucinatory AR, inspired in part by Surrealist artist Max Ernst. In “What Is the Mechanism of Collage?” (1936), Ernst writes:

One rainy day in 1919, finding myself in a village on the Rhine, I was struck by the obsession which held under my gaze the pages of an illustrated catalogue showing objects designed for anthropologic, microscopic, psychologic, mineralogic, and paleontologic demonstration. There I found brought together elements of figuration so remote that the sheer absurdity of that collection provoked a sudden intensification of the visionary faculties in me and brought forth an illusive succession of contradictory images, double, triple, and multiple images, piling up on each other with the persistence and rapidity which are particular to love memories and visions of half-sleep.(17)

Of particular interest to my work in exploring and experimenting with “Hallucinatory AR” was Ernst’s description of an “illusive succession of contradictory images” that were “brought forth” (as though independent of the artist), rapidly multiplying and “piling up” in a state of “half-sleep.” Similarities can be drawn to the process of the seemingly disparate AR images in Hallucinatory AR, jarringly coming in and out of view, layered atop one another.

One wonders if these visual accidents are what the future of AR might hold: of unwelcomed glitches in software systems as Bruce Sterling describes in Beyond the Beyond at Wired.com;(18) or perhaps we might come to delight in the visual poetry of these augmented hallucinations that are “as beautiful as the chance encounter of a sewing machine and an umbrella on an operating table” (Comte de Lautreamont’s often quoted allegory, famous for inspiring Surrealist artists Max Ernst and Andrew Breton).(19) To a computer scientist, these “glitches,” as applied in Hallucinatory AR, could potentially be viewed or interpreted as a disaster, as an example of the technology failing. To the artist, however, there is poetry in these glitches, with new possibilities of expression and new visual forms emerging. Further to the theory discussed in this paper, the “Hallucinatory AR” artwork was not a traditional form, but a new approach, allowing the medium of AR to inspire and evolve new forms and modes of expression, “tapping into the unique qualities of expression the new form enables.” Further, in regards to Méliès introducing new formal styles, conventions, and techniques that were specific to the medium of film, novel styles and new conventions will also emerge from AR artists. Méliès became famous for the stop trick, or double exposure special effect, a technique which evolved from an accident: Méliès’s camera jammed while filming the streets of Paris; upon playing back the film, he observed an omnibus transforming into a hearse. Rather than discounting this as a technical failure, or “glitch,” he utilized it as a technique in his films. Hallucinatory AR also evolved from an accident, which was embraced and applied in an attempt to evolve a potentially new visual mode in the medium of AR.

Access for the Artist

Pearson states: “Electronic tools are still resistant and clumsy and do not facilitate working in a direct and expressive manner.”(20) He writes of a more direct involvement of the artist in traditional art media. Pearson comments on his own “love/hate relationship with the computer,” commanding too much of his time to learn “its language” and his “resistance to learn programming.”(21) Issues of access are key to artists’ creative investigation of a medium and will be discussed further in regards to Malina.

This returns us to questions of access for the artist, to which, like Pearson, Malina notes, “the computer was not developed with the specific needs of artists in mind.”(22) Margot Lovejoy notes how the “first wave” (1965–75) of computer use in the visual arts was dominated by scientists with access to equipment with larger number of artists beginning to gain access in the “second wave,” then the next decade continuing with pioneer artists who participated in developing new software tools. She notes that as artists continue to gain access to powerful computing tools at a lower cost, a new aesthetic begins to emerge. This echoes Gene Youngblood’s belief in Martin Lister’s New Media: A Critical Introduction, that: “The full aesthetic potential of this medium will be realized only when computer artists come to the instrument from art rather than computer science.”(23) I agree, and we are beginning to see this as artists start to work with the emerging technology of AR, which traditionally belongs to the realm of Computer Science. In regards to Lovejoy, we can think of AR as being in its “second wave” currently. New aesthetic and stylistic tendencies will emerge as tools become more accessible and easy for artists to use, ideally developed with and by artists.

I am intrigued by how Douglas Rushkoff’s book, Program or be Programmed,(24) pertains to artists as nonprogrammers working in AR. My view is that as typically nonprogrammers in AR, artists need to dive into the technology to understand the rules, then try to bend and break them. However, in order to truly advance a new medium, must the artists keep up with the programmers? My approach has been one where I work with constraint and find ways to repurpose the software as it exists and extend this in creative ways. It has also been my experience to work and collaborate with programmers, as I believe my strength lies in the creative application as opposed to the custom coding of the software. I work as directly with the software as I can, tinkering, and even breaking the technology to understand how it works (I discuss this in my TEDx talk, “How Does Wonderment Guide the Creative Process?”(25)); but, as artists, do we need to go even further beyond the user-friendly tools and preset functions available? Certainly, there are artists who are also excellent programmers and technologists. To this distinct group, yes, they have the ability and skill set to do this. Otherwise, in addition to tinkering, and learning the technology as well as possible, dependent on each person’s unique skill set, I believe the answer lies in collaborations and dialogues between artists and programmers.

An Artist’s Approach to Technology as a Tool or Medium

Digital artist and new media theorist Hector Rodriguez differentiates the ways in which an artist can approach technology: as a tool or as a medium. Rodriguez states: “Media artists who view technology as a tool would often prefer to keep it invisible.”(26) Artists who, in contrast, view technology as a medium typically prefer to: “Emphasize its visibility and thus resist all pressures to hide the internal operation of digital media.”(27) The artist who approaches technology as a tool applies it “to realize a prior idea” where the outcome is “predefined and predictable” with the artist not typically “acquiring an in-depth understanding of the technologies used in their work.”(2)8

When technology is treated as an artistic medium, creative ideas are drawn from an in-depth knowledge of how technologies operate. (This relates back to my approach to AR as discussed above.) Rodriguez gows on: “Media technologies thus become sources of artistic ideas, not mere conduits for their implementation. The creative concept is essentially dependent on a thorough understanding of the technologies that will actualize it.”(29) We can also look back to Méliès here with the technology being the source of inspiration, of story evolving from technique.

Malina writes: “Several lines of analysis are needed to elaborate the new kind of art forms that are enabled by the computer. The first involves understanding the specific capabilities of the computer and creating art forms that exploit these.”(30) Malina notes how this approach, “experimental and empirical,” is followed by several computer artists. I would position this approach to technology as a medium by Rodriguez’s definition, where art forms and creative ideas emerge from a thorough knowledge and understanding of how the technology works. Malina, like Pearson, however, warns that artists can fail to transcend the capabilities of the technology, with advocates of the empirical approach arguing that “until the artist has access to the technology, its potential for art-making cannot be fully understood.”(31)

Marshall McLuhan’s Tetrad and AR

Adapting McLuhan’s Tetrad to AR is a valuable exercise in analyzing prior forms that AR has evolved from and where it may go. McLuhan’s Tetrad consists of the following media effects: enhance, obsolesce, retrieve, and reverse. Let us examine AR as it pertains to each of these effects. What aspect of society does AR enhance or amplify? I believe AR enhances information sharing, entertainment, gaming, education, and human vision. What does AR “obsolesce” or push aside? I believe AR obsolesces Virtual Reality and virtual intangible environments. What does AR “retrieve” and pull back into center stage from the shadows of obsolescence? I believe AR retrieves tactility, physical engagement, mobility, and physical space, as well as the single-user looking device (for example, early Victorian media such as the stereoscope). What does AR reverse or flip into when the medium has run its course or been developed to its fullest potential (looking ahead to the future of AR)? I believe AR reverses the window and screen where there is no identifiable filter/mediation and the individual is no longer able to distinguish reality from the virtual. AR also has the potential to reverse into advanced sensory technologies where the brain is linked to the digital realm in a scenario where there is a direct interfacing of the brain with the world.

Remediation and Transition

For the artist to truly create something novel and aid in fostering the creative growth of an emerging technology, the artist must tap into the unique qualities of expression the new form enables, rather than reverting to previous forms. In Remediation, media theorist Steven Holtzman argues that approaches of repurposing “do not exploit the special qualities which are unique to digital worlds” and that “it is those unique qualities that will ultimately define entirely new languages of expression.”(32) He describes repurposing as a “transitional step” that permits “a secure footing on unfamiliar terrain,” and is quick to note that this “is not where we will find the entirely new dimensions of digital worlds.”(33) Holtzmann urges that we must “transcend the old” to discover the new for, “like a road sign, repurposing is a marker indicating that profound change is around the bend.”(34) I fully concur with Holtzman.

The present is a transitional time in AR; novel forms, styles, and conventions are just around the bend. It is a critical time for artists to experiment with the technology and act as pioneers to help shape the medium, harnessing the unique capabilities of AR to generate new modes and techniques.

Footfalls, by Tmema [Golan Levin & Zachary Lieberman] (2006), is an interactive audiovisual installation in which visitors’ footsteps create cascading avalanches of leaping virtual shapes.

Footfalls, by Tmema [Golan Levin & Zachary Lieberman] (2006), is an interactive audiovisual installation in which visitors’ footsteps create cascading avalanches of leaping virtual shapes.

The Importance of Artists Working with New Technologies

New media artist Golan Levin discusses how new media artists have early on prototyped many of today’s technologies. He writes:

As an occasional emissary for new-media arts, I increasingly find myself pointing out how some of today’s most commonplace and widely appreciated technologies were initially conceived and prototyped, years ago, by new-media artists. In some instances, we can pick out the unmistakable signature of a single person’s original artistic idea, released into the world decades ahead of its time—perhaps even dismissed, in its day, as useless or impractical—which after complex chains of influence and reinterpretation has become absorbed, generations of computers later, into the culture as an everyday product.(35)

Levin states the importance of “including artists in the DNA of any serious technology research laboratory (as was practiced at Xerox PARC, the MIT Media Laboratory, and the Atari Research Lab, to name just a few examples)” since “the artists posed novel questions which would not have arisen otherwise.” Levin urges that in order to “get a jump-start on the future,” it is necessary to “bring in some artists who have made theirs the problem of exploring the social implications and experiential possibilities of technology.” He continues: “What begins as an artistic and speculative experiment materializes, after much cultural digestion, as an inevitable tool or toy.”(36)

Therefore, the role of the artist in the early stages of an emerging technology is more important than one may think, with such artistic explorations leading to commonplace technologies, as Levin states. The role of the artist in working with emerging technologies in general is to be inspired by technology as a medium (but not be blinded by it), to understand its capabilities, and to exploit them as a “colonizer” and a pioneer, and not to fall prey to remediation but, instead, to strive to create an entirely new language of aesthetics and forms, which may even entail breaking out from the existing conditions of the environment. Perhaps this is the only way to truly create something new and innovative. This is where we need to go with AR, and it is imperative that AR artists serve as pioneers in guiding us there.

Notes

1. John Pearson, “The computer: liberator or jailer of the creative spirit,” Leonardo, Supplemental Issue, Electronic Art 1 (1988): 73.

2. Ibid.

3. Ibid., 78.

4. Ibid., 76.

5. Ibid., 79.

6. Ibid.

7. Marshall McLuhan, “The relation of environment to anti-environment,” in Marshall McLuhan Unbound, Eric McLuhan and W. Terrence Gordon (eds.). (Corte Madera, CA: Ginko Press, 2005), 9.

8. Roger F. Malina, “Digital image—digital cinema: the work of art in the age of post-mechanical reproduction,” Leonardo. Supplemental Issue, Digital Image, Digital Cinema: SIGGRAPH ’90 Art Show Catalog 3 (1990): 34.

9. John Andrew Berton Jr., “Film theory for the digital world: connecting the masters to the new digital cinema,” in Leonardo: Journal of the International Society for the Arts, Sciences and Technology. Digital Image/Digital Cinema: A special edition with the Special Interest Group in Graphics (SIGGRAPH) Association of Computing Machinery (New York: Pergamon Press, 1990), 7.

10. Ibid.

11. Ibid.

12. Pearson, “The computer,” 76.

13. Dan North, “Magic and illusion in early cinema,” Studies in French Cinema 1.2 (2001): 70.

14. Pearson, “The computer,” 76.

15. Blair MacIntyre, Maribeth Gandy, Steven Dow and Jay David Bolter, “DART: A toolkit for rapid design exploration of augmented reality experiences,” in Conference on User Interface Software and Technology (UIST’04), October 24–27, 2004, Sante Fe, New Mexico, 197.

16. Helen Papagiannis, “‘Wonder Turner’ and ‘The Amazing Cinemagician’ augmented reality and mixed reality art installations,” 2010 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)—Arts, Media, and Humanities (ISMAR-AMH), Seoul, South Korea, Oct 13–26, 2010, 27–32.

17. Max Ernst, “What is the mechanism of collage?” in Theories of Modern Art, Herschel B. Chipp (ed.) (Berkeley: University of California Press, 1974), 427.

18. Bruce Sterling, “Augmented reality glitches,” Beyond The Beyond, August 29, 2009.

http://www.wired.com/beyond_the_beyond/2009/08/augmented-reality-glitches.

19. Robert Williams, Art Theory: An Historical Introduction (Malden, MA: Blackwell Publishing, 2004), 197.

20. Pearson, “The computer,” 78.

21. Ibid., 80.

22. Malina, “Digital image,” 34.

23. Martin Lister, New Media: A Critical Introduction (New York: Routledge, 2009, 2nd edition), 60.

24. Douglas Rushkoff, Program or be Programmed: Ten Commands for a Digital Age (New York: OR Books, 2010).

25. Helen Papagiannis, TEDxYorkU presentation: “How does wonderment guide the creative process?” (October 2010), http://www.youtube.com/watch?v=ScLgtkVTHDc.

26. Hector Rodriguez, “Technology as an artistic medium,” 2006 IEEE International Conference on Systems, Man and Cybernetics, Taipei, Taiwan, October 8–11, 2006, 3638.

27. Ibid.

28. Ibid., 3635.

29. Ibid.

30. Malina, “Digital image,” 34.

31. Ibid.

32. Jay David Bolter and Richard Grusin, Remediation: Understanding New Media (Cambridge MA: MIT Press, 1999), 49.

33. Ibid.

34. Ibid.

35. Golan Levin, http://www.flong.com/blog/2009/new-media-artworks-prequels-to-everyday-life.

36. Ibid.