1 Introduction

This paper considers the case of contemporary artworks in Augmented Reality (henceforth, AR) and Mixed Reality (henceforth, MR). In the last ten years, as a consequence of the increasing diffusion of virtual technologies in everyday life, artists have begun to express themselves through these media, adapting to their evolutions and contributing to the rethinking of their development and of their possible concrete applications. More and more, AR and MR art seeks to answer the question: “how to create AR with an impact on our physical reality?” (Veenhof 2018: 354).

Despite the fact that our world results from the merging of digital and concrete elements, virtual objects are often considered non-real-entities of a delusional (and therefore dangerous) nature. In this article I first analyse recurring narratives about AR and MR technologies that depict them as a disturbing presence. I will suggest considering these descriptions, frequently present in movies, tv-series, and artworks, as based more upon a dystopian conception concerning the future than on an understanding of the current use of virtual technologies.

I will then show some relevant case-studies, selected from actual contemporary AR and MR artworks. I will propose considering them as “walking artworks” characterized by three main features: they are installed throughout the city, reconfiguring the usual meaning of some places; they are conceived to be traversed and explored in first person by the users, who are usually engaged in a creative performance, which is for the most part improvised and gestural; they highlight the technical, multimedia, and interactive character of human imagination.

2 A Narrative of Augmented and Mixed Reality

Variously imagined since the 1980s in science fiction books and movies, video games and comic strips, AR emerged in the 1990s, largely thanks to experiments with Head-Up Displays (Gatti 2019). These devices, used mainly in military applications, overlay additional information onto the user’s visual field in a way that still allows her to perceive her surroundings. The term AR refers to all those technologies that, like Head-Up Displays, involve the superimposition of virtual elements on the concrete world, enhancing the subject’s operational and perceptive possibilities and expanding her experience of the environment (Liberati 2016). These could include texts and information available on a mobile phone or 3D objects placed in the usual action space. There still is no agreement on a precise definition of AR. Some scholars believe that when we interact with three-dimensional and complex digital objects that are integrated into the everyday world we should speak of MR, rather than AR, since MR is often held to include a blurring of the concrete-digital distinction (Arcagni 2018). However, in many cases, leading companies such as Apple, Adobe, and Microsoft use the two terms interchangeably, at times even when referring to the same tool, context or project.¹

From its very origin, AR was conceived not to replace, but to integrate, modify and enhance the real environment with electronic functions, adding them to the affordances already present in it (Wellner, Mackay, and Gold 1993). In that sense, it is possible to consider MR as the radicalisation of AR, since MR is not defined by the complexity of the digital objects embedded into the environment but rather by its tendency to remove the boundary between the concrete and the digital. Today, AR is mostly developed for smartphones and other mobile devices; its interface takes the form of a virtual image, and there still is a clear frame defining it. On the other hand, there are already devices like Microsoft Hololens 2² and Oculus Quest 2³ that do not necessarily involve the use of controllers; with these, the user freely interacts with three-dimensional holograms which appear alongside concrete objects.

In general, I believe that what is common to all these technologies is a strategy of digital enhancement and an extension of reality through virtual elements. Following Andrea Pinotti (2017), virtual digital images present themselves as an-icons, namely as environments or parts of them, negating their iconic and representational nature. Given their increasingly widespread diffusion in the everyday environment and their tendency to blur the threshold between the real-world and the image-world, these images can elicit a strong sense of presence. Being pictures on a screen or holograms, they are manipulable entities through which users intervene in the world and interact with others. In my opinion, even if they are virtual objects they have real and transformative consequences on the users’ lives. They thus reconfigure the visual, which becomes “a multisensory, interactive, responsive, ‘autonomous’, ‘intelligent’, or at least ‘smart’ universe” (Arcagni 2018: 119). The user is no longer merely an “observer” but an “experiencer” (Pinotti 2017: 1); these images are operative interfaces, tools that actively mediate the relationship between human beings and the outside world (Hoel 2018), allowing, to different extents, for interaction with and manipulation of the environment (Fedorova 2015). As Trevor Paglen pointed out, they can be traced back to the operational images Harun Farocki explored from the early 2000s: they are images that do not just depict something, they “do things” (Paglen 2014).⁴

The contemporary world is already mixed, because it results from the integration of various kinds of materiality, both concrete and virtual. However, the digital enhancement of the environment is often interpreted as a disturbing irruption of a phantom presence, one that threatens to take a dystopian turn in the near future.

There are already many artists who are highly critical of AR, mainly concerned with its ability to delude the user about the reality of digital objects. For instance, Magali Barbé’s short film Strange Beasts (2017) tells the story of Victor, game designer and originator of the AR game Strange Beasts, which allows users to create friendly-looking monsters to their liking. Strange Beasts harnesses a technology which, through the use of specially-designed contact lenses, projects these odd creatures directly onto the user’s retina. Victor enthusiastically underlines the interactive and playful nature of this new and revolutionary type of entertainment. He shows his daughter having fun with characters generated straight from her imagination. However, as the viewer soon realises, Victor is so immersed in his augmented world that he has completely lost touch with reality. The little girl he plays with is as digital as the other strange beasts; seen from the outside, Victor is a lonely man speaking with invisible entities.⁵

Another artist, Keiichi Matsuda, film-maker and designer of Augmented, Mixed and Virtual Reality projects, has also been reflecting for years on the problems that could follow from the dissemination of these technologies. In his famous film HYPER-REALITY (2016), the viewer’s gaze coincides with that of a woman wearing an AR device: she moves with difficulty across her city, her visual field covered in hundreds of pop-ups featuring information, advertising and entertainment. Sounds and images fill the perceptual field, all the woman’s activities are linked to a personal account through which she can earn points, on the bus, in the supermarket, even at a church. The hyper-performance technology produces a paradoxical contraction of the subject’s perceptual possibilities. Technology is no longer at the service of the individual, but rather it diminishes the subject’s interpretive creativity and impoverishes her social relations.⁶ Likewise, in his most recent 360^o movie Merger (2018), the protagonist, a young accountant, cannot match the optimisation capacities of technology and decides to undergo an operation that will allow her to merge with the network, sacrificing her own existence to better satisfy her clients.⁷

This kind of artwork is important because it reflects on the design of digital interfaces. It raises questions about the need for regulation concerning users’ privacy, and the importance of safeguarding them against a framework directed towards a logic of consumption.

In some respects, however, I consider these criticisms too as problematic: while a virtual child will never have the same characteristics of a real child, this does not mean that digital objects are not real. Although Barbé’s pixel child is real, she is not truly a child, just as her exact reproduction in wax or plastic would not be. Whether virtual elements are fictional or not is not so much a matter of their being digital, but rather of their properties and their function in the world (Chalmers 2017). Since they are not concrete, there are many needs that virtual elements cannot satisfy, from affective to physiological ones, but this seems to me to be precisely why we do not risk losing contact with things.

Matsuda’s analysis, which is in some respects less catastrophic, is surely more effective because it predicts a more plausible scenario. Nevertheless, there are some unjustified exaggerations in his work too. While it is unlikely that the virtual labels and objects superimposed on the environment will all be activated simultaneously, because this would lead to the complete ineffectiveness of the technical tool, the protagonist’s frustration with technology also stems from her using it in a mistaken way. The woman asks the Google Assistant questions such as “Who am I?” and “Where am I going?,” which the technology, interpreting them literally, inevitably fails to answer, providing instead biographic facts and navigational routes on Google Maps. The result is a portrayal which demonises the digital as something that infests our lives without in any way contributing to or improving them. The digital and the virtual completely miss the essence of the human; they distort and annihilate it. Thus, Matsuda seems to allude, not too subtly, to a distancing away from a more authentic — if not de-technicalized, at least de-digitised — human dimension. 

Imagining the future is useful and necessary. However, we must not forget that these are not just futuristic technologies. They are already variously and widely used in fruitful ways. Their present application is worthy of analysis. The next section examines the work of artists who currently employ AR not only as a subject of study but also as a medium of reflection and narration.

3 Narrating with Augmented and Mixed Reality

Since the first experiments after its emergence around 2010, AR and MR art has tended to integrate with urban space. In 2011, the political art collective Manifest.AR took part in the Occupy Movement with hundreds of AR protest artworks in the area surrounding the New York Stock Exchange.⁸ Thus, a hybrid space for dissent and artistic creation emerged: “the Safety Glass of the Display is shattered and the Physical and Virtual are united in a new In-Between Space.”⁹

Manifest.AR is generally recognized as the first artistic collective that started to augment the physical environment with virtual elements in its interventions (Geroimenko 2018). It is true, however, that other new media art collectives, such as the British Blast Theory, had already been experimenting with interactive online technologies since the early 1990s. Blast Theory creates actions that ask for the active involvement of the public in the urban space. Doing so, they explore the political implications of the wide use of the Internet in the city space, and the way it mediates relations with other individuals. For example, in I like Frank (2004), an adventure game which took place in Adelaide, users were playing both in the streets and online, comparing experiences and sharing information by means of a mobile phone to find a fictional character named Frank.¹⁰

As frequently happens in Blast Theory performances, AR and MR artworks often take place in public spaces, becoming a tool for stressing the potential of new technologies, and understanding their social and political consequences. However, in contrast with other new media artworks, these stratify the concrete space by adding to it new virtual information and objects, only visible to those who use an AR or MR device such as a smartphone, a headset or glasses, being physically present at the place.

QR codes were one of the first and cheapest forms of augmenting reality, used in particular by street artists. For instance, from 2011 to 2017, the street artist Sweza placed his QRadio, a stencil shaped like a stereo and featuring an embedded QR code, in various cities around the world. Once captured with a mobile phone, the QR code activated an audio track, connected to the physical place like a sort of a soundtrack.¹¹ Street art was thereby enriched with sound elements aimed at transforming the perception of urban space. 

In institutional contexts, too, both group exhibitions and individual AR and MR artworks have gradually taken the form of itineraries through city spaces. In 2019, the New Museum of New York, in collaboration with Apple, funded six works by Nick Cave, Nathalie Djurberg and Hans Berg, Cao Fei, John Giorno, Carsten Höller, and Pipilotti Rist for the [AR]T project. In various cities worldwide, such as Tokyo, Paris, and New York, visitors could participate for free with their mobile phones. The artworks formed an “interactive,” “experiential” and “free to the public” walk.¹² In some cases, they asked participants to track specific objects, such as the poetic lines of Now at the Dawn of My Life by John Giorno or the coloured shapes of International Liquid Finger Prayer by Pipilotti Rist, which appeared gradually as the visitor advanced along the path, framing the surroundings with a smart device. In others, they were filters through which the environment could be viewed in a different way, as in the case of Through by Carsten Höller, which made it possible to see the surroundings with the sense of depth removed.

Likewise, for the more recent Unreal City (December 2020-January 2021), the Acute Art app and Dazed Media produced a tour along the river Thames made up of 36 AR and MR artworks by Olafur Eliasson, Cao Fei, and Tomás Saraceno, among others.¹³ As with QRadio and [AR]T, the artists invited visitors to search for hidden virtual entities which would alter their usual perception of the space. In general, Acute Art makes it possible for users to see some of the artworks wherever and whenever they want. They can install them in their own homes, or in other everyday contexts, allowing the works to take on different meaning each time.

In 2019, the Serpentine Galleries in London also produced a series of artworks which integrated physical reality and virtual elements. Jakob Kudsk Steensen’s The Deep Listener, notably defined as a “journey” through Kensington Gardens and Hyde Park, which provides an opportunity for users to explore the lives of five plant and animal species through their sights and sounds, presents itself as both a site-specific work and a digital archive.¹⁴ As the title suggests, by photo-capturing specific objects spread through the gardens, the visitor can hear sounds which are not ordinarily audible, activated together with texts and virtual objects that are superimposed on the concrete environment.

Hito Steyerl’s Actual Reality^os app worked similarly: once downloaded onto a mobile phone, it was activated by capturing various markers around the Serpentine building.¹⁵ With the app running, users walking around the museum and capturing the markers would see the building distorted according to the representation of information about the local community’s health, working, and living conditions, making immediately visible the real impact of social phenomena. The Serpentine’s projects build a complex narrative, employing texts, audio and video tracks, official data and personal testimonies to bring out and make perceptible information that would otherwise be hidden. AR offered Steyerl the opportunity to work at that kind of “imperfect” artwork estimated by the Cuban film director Julio García Espinosa in his For an Imperfect Cinema manifesto (1969), one that “diminishes the distinctions between author and audience and merges life and art” (Steyerl 2009). While Espinosa looked at new media technology at the end of the 1960s with enthusiasm, as the promise for finally realizing an art of the people, Steyerl openly asserts the ambiguous character of viral online images (ibid.) and of digital technologies (2017), both in the art world and in the everyday world, trying to show that they are both a medium for expression and a tool for social contempt, control and repression. Actual Reality^os represents one of the well-known German artist’s and theorist’s many efforts to employ new technologies with an activist intent, to construct a shared strategy for denouncing and solving problems and for talking about a place from the inhabitants’ point of view.

I would argue, then, that a particular type of artwork is thus taking shape — one that harnesses the virtual augmentation of the everyday environment to tell stories about places and the people who inhabit them. Both concrete and digital components are part of the work, which offers the user a space to explore actively from a haptic, auditory, and visual perspective.

In this kind of operation, personal and collective memory are transformed into spatial storytelling, as in Emilio Vavarella’s MNEMOSCOPIO (2020).¹⁶ Produced in Capo di Leuca (Italy) the work consists of a 3D map and an audio track superimposed on physical reality and perceived through a specially-modified helmet designed by the artist. The map and audios were created from the memories of those who had lived abroad for several years and then returned to their places of origin. The surrounding environment is also digitized and thereby remediated by the helmet.¹⁷ One’s own experience and that of others, as well as the present and the past, reality and imagination, the human and the technological gaze overlap and generate a perceptive short circuit. In Vavarella, hence, the focus on travel, on integrating the concrete and the digital, and on narration return as an experience to be lived in the first person.¹⁸

The unique spatial storytelling of AR and MR artworks is therefore particularly suited to the revitalization of the places and contexts of the past. Unlike a book or a movie, the addition of digital elements allows the story to be grounded in specific contexts. At the same time, it is not just a guided tour aimed at providing information, for the narrative is presented in a form that is perceptible by everyone (Wither et al. 2010). For example, in 2019, the Italian cultural association Landworks launched ARgentiera in Augmented Reality, which involves a series of residencies for the production of site-specific AR artworks to tell the story of the community of the Argentiera Sardinian mine, which has become an open-air museum.¹⁹ Visitors are free to move through the old village streets, choose between the routes within it, get intrigued by artists’ drawings and photos, and capture them with their mobile devices to complete the work, activating digital animations.²⁰

Beyond the specifics that characterize them, some elements recur in each of the examples mentioned so far. I propose to consider them as “walking artworks”: they are designed to be traversed on foot and explored since they are integrated into the city environment’s space and time. The merger of physical space and virtual elements makes it possible for a particular type of multi-sensory and interactive storytelling to arise.

First of all, then, as a function of virtual augmentation, in walking artworks the environment acquires new levels of significance, which inevitably modify the experience of the context. Familiar places are thus invested with new meanings only available for those who use the required technology.

Artists use virtual elements to bring about concrete changes in the space in which the work is installed. Whether they reconstruct the past or imagine the future, walking artworks creatively reconfigure the world. Their tendency to add to the environment follows a practice of augmentation that was already present in many forms of new media artworks, realized thanks to other kinds of technology. The long series of Janet Cardiff audio walks across different cities; the augmented sculptures of Pablo Valbuena, projected onto the surface of buildings; or the data paintings of Refik Anadol: all reveal some hidden information linked to a specific place. The laser, LED and steam installation Waterlicht (2015) by Studio Roosegaarde, which gave the impression of being below the rolling sea under which Holland, without its dams, would be submerged, exemplifies the capability of using simple technology to provide site-specific knowledge (Mancuso 2018).²¹

Employing AR and MR technology, walking artworks owe much also to video games in general and to AR video games in particular. As in a video game, the visitor moves inside the narration and guides it according to her actions to complete a series of missions. In the case of walking artworks, the story itself is the reward. Geocaching, iButterfly, and especially Ingress and Pokémon Go have paved the way for the overlapping of complex virtual elements on the physical environment, resulting in its resemantization. Although their objects are virtual, they are still real, for they have a concrete effect on the lives of both players and of non-players who share the same action space (Liberati 2018).

Secondly, in walking artworks artists ask the visitor to contribute actively. They engage her by involving her body in different perceptual modalities, leave her time to dwell on different details at will, provide the opportunity to take photographs, and also to modify and share them. The viewer can decide to extend or shorten her stay; ignore some paths to favour others; and even leave out certain elements of the story altogether. Visiting the same walking artwork is never having the same experience twice: each time you can make different choices, interact differently with the virtual entities (sometimes also manipulate and install them), improvising gestures and movements. If the contemplation of an image is never an entirely passive task, in this case, as in many performative pieces of participative theatre,²² the visitor cannot avoid participating if she wants to see the complete artwork. In walking artworks, she is called to lose herself in the narration, like a sort of flâneur, and to become “aware of a new urban dimension precisely by following the path of the artworks” (Biggio 2020b: 99).

This kind of artwork elicits a strong sense of presence, since it relies on the interactivity, agency, and mobility of the individual’s body, which does not have to be reproduced digitally and is free to move around the physical environment. The technological device being used does not in itself command attention; it becomes a vehicle for observing and perceiving the world in an unprecedented way. It is not the object to look at, but one through which to perceive the surrounding space, as if it were a “magic lens” with transformative powers (Wither 2010: 45). The participant feels immersed because the artwork makes her responsible for the story’s gradual progression by appealing to her conscious involvement.

While, as already shown, interactivity, integration of the real and the virtual, and the viewer’s freedom to control the point of view are not, taken separately, exclusive features of AR and MR artworks, it is only here that they all are present simultaneously, transforming the visitor into a moving reader (Koenitz 2015). The contemporary hyper-reader is accustomed to juggling multimedia contents that refer to different temporalities and to contributing to their production — not only because she offers different interpretations of a story but also because she follows different paths within it, possibly adding elements and sharing her experience with others, remotely or in person (Valdivieso 2020).

As Federico Biggio pointed out, AR is “a mirror that reveals truths about the subject that is mirrored” (2020a: 107), with which one can construct and in which one can recognize one’s own “data-representation of the world” (2020a: 106). AR is not only a technology that generates new affordances and reveals information hidden to the individual, but first and foremost one that reflects the ways that human beings interact with the outside world. Indeed, I believe it underscores their constant tendency to enhance the experience of the world with “virtual” and manipulable elements. In observing the environment and its objects, the human subject immediately interprets it according to its operational potentialities. She projects herself into space and time to build possible-action scenarios. Thanks to her imagination, she both places herself in objects and extends her body with things. The imaginative power of projecting oneself virtually toward the outside in a different time and space reveals “technics not as the product of culture over nature, but rather as the expression of an aesthetic structure of the human mind-body system” (Dalmasso 2019: 82), which constantly tends to expand and modify itself with technical objects. Humans are not the only animals that make tools. Nonetheless, what distinguishes them from other species is the feedback that technical objects have on their cognition, intervening and modifying it (Ihde and Malafouris 2019).

Thus, thirdly, AR and MR allow the technical-operational character of the human being’s imagination to emerge (Hansen 2006), since they make its activity and products perceptible. In particular, Lambert Wiesing holds that “non-immersive virtual reality emerges from an assimilation of the image object to the imagination” (2010: 89) because it allows for a similar manipulation of the picture, which also negates its iconic character because it is always open to change. This manipulation is certainly not without limits, all the more because a completely free imagination would not really be engaged with the outside world. Human imagination is material since it reconstructs the past and designs the future while interacting with the present world (Malafouris 2020); it interfaces constantly with its surroundings and runs up against its characteristics (Montani 2014).

AR and MR technical images are as metastable as the contents of the individual’s imagination, but they give to them a concrete form (cf. MNEMOSCOPIO), which can thus be perceived, shared, manipulated and contaminated by the community in different ways and to varying degrees. In most of the artworks mentioned above users cannot add new contents.²³ On the other hand, in almost all cases, users are free to take a screenshot or a selfie of themselves in the artwork, to modify the image and share it with others in private messages or on social networks. The work is released from its creator’s intentions and comes to life in other contexts.

In the examples discussed here, as in many other cases, AR and MR art accepts, and indeed requires, the collective construction of a shared work, which often takes the form of a narrative. Since these are works that gather information and testimonies linked to the context in which they are installed, the user is frequently confronted not only with the result of the artist’s imaginative effort but also with that of the local community, transformed into an environment to be traversed.

Indeed, storytelling in AR and MR makes use of what Papineau (2016) calls the natural tendency of human beings to construct a narrative on the basis of their own experience of the environment, merging information acquired first-hand with personal inferences and the testimonies of others. These usually involve varied and complex perceptual information, which respects neither a spatial nor a temporal order. The individual arranges such information so that she can understand it. Compared to other forms of narration, the multimedia and interactive character of AR and MR storytelling better matches the kind of perceptual material the individual confronts daily. In this sense, it does not require the participant to follow a predetermined course of the story, but provides her with a variety of viewpoints from which to choose. The experience of others is put to the service of the individual; it becomes part of her personal history. Pietro Montani (2014) recalls how Wolfgang Iser describes reading literature as the only kind of practice in which the subject’s point of view moves within the object being explored. It seems that virtual technologies have produced a new form of this kind of narrative. Reading is always a formative experience, for it modifies the reader’s knowledge about the world. In this case, however, the reading takes place through direct experience of the world.

Contrary to what might be the popular view, in AR and MR artworks produced to-date, digital enhancement does not aim to saturate and muddle the perceptual field. When the work intends to deceive the viewer, it does not do so malevolently, but so as to produce an aesthetic effect, as has already been the case with in various works not created digitally (Conte 2020). As Ronald Azuma (2015) points out, AR storytelling establishes meaningful connections between the virtual and the concrete, generating results less “virtual” than other, more classical forms of storytelling, such as in books, TV series, films, or video games that do not display in the physical environment. 

The variety of stimuli AR and MR offer can even be an asset for developing critical thinking. Unlike Virtual Reality (VR) installations, which direct the individual’s experience with limited degrees of freedom, AR and MR leave the participant’s imagination free to make associations between elements of different natures (past and present; digital and concrete), arranging them into different and creative levels of interpretation and action — ones not necessarily prefigured by the artist. For instance, discussing the renowned virtual installation Carne y Arena by the Mexican director Alejandro González Iñárritu, Luca Acquarelli (2020) shows how the VR installation was conceived as part of a more complex exhibition path made up of six rooms: five physical spaces and the actual virtual environment.²⁴ Acquarelli believes that the artwork’s political character emerges from the multimedia composition of the work considered as a whole. Only by assembling various types of images, devices, and materials is it possible to build a complex, valuable, and non-stereotypical testimony of something happening in the real world. In the VR environment, the viewer’s imagination is not given much freedom due to the reduced degree of interactivity. The participant, who wears a helmet, cannot see her body, cannot grasp objects, cannot intervene in the story or interact with the characters.²⁵

On the contrary, the specificity of AR and MR storytelling relies on a multimedia construction of the work and on a high degree of interactivity, appealing to individual agency and responsibility.

4 Conclusions

The human need for technology is a structural one, far from representing an evil dismissal of its supposed “natural” and “genuine” origin. Between human and technical being there is a relationship of interdependence which Pietro Montani has called “empowerment” (2017): there do not exist separate and independent technical and human bodies, with one set juxtaposed to the other. The innervation in material culture allows human cognition to express itself creatively with and through technological objects, including digital tools.

I consider technologies like AR and MR as a particularly fertile ground for the imagination’s interactive tendency. In particular, as I’ve shown, AR and MR artworks narration endeavours to construct a world, modifying the everyday environment with new objects and envisioning a future version of it, thanks to the work of a plural imagination. Walking artworks are not the only type of AR and MR artwork. Nevertheless they certainly constitute a recurring schema in the works that use these technologies. In fact, as I have tried to show in this article by means of several examples, they tend to take place across the urban space, calling for the active and creative interaction of visitors, asking for their participation in order to be completed.