1 Introduction

The monographic section of this issue of Cinergie arises from a need that has becoming increasingly acute in the past few years, namely that of closely examining the modes, practices, strategies, and forms of storytelling used in the various forms and experiences of Virtual Reality (VR) entertainment.

The question that we asked ourselves, and thus also the field of analysis that we wanted to open focuses primarily on two elements: first, on if and how cinematic narration can exist in VR; second, on what paths the development of a form of storytelling not directly tied to cinematic language might take.

The question of cinematic VR production has been on the table for several years. This is due to the peculiarity of VR language which, even if it is defined by an image that surrounds and immerses the viewer rather than placing them, as in the classic cinematic situation, in front of a screen, relies decisively on an audiovisual basis that cannot help but refer to cinematic practices of constructing visual and auditory experience. Despite this, it would be extremely reductive to consider VR as the mere transposition of elements of cinematic language. The VR medium is endowed with its own specificity, which inevitably impacts its forms of narration. We thus need to investigate the narrative forms it uses that are probably related to cinematic language, and draw their strength from the same basis, drink from the same well, but develop according to different trajectories, thus displaying different links and affinities.

In this introduction we will aim to outline some of the principal transformations that VR is imparting to audiovisual language and experience, with a particular focus on the impact of these innovations on storytelling. The essays of the section span a variety of theoretical approaches and backgrounds, argumentative styles and case studies (e.g. documentaries, games, art), but all of them share a similar rhetorical attitude: underscoring the potential and novelty of VR but, at the same time, identifying a series of remediations and limitations. In particular, they will show that most of the contemporary discourses on VR need to be reframed critically in order to reflect more objectively on complex notions and dynamics such as screen, interactivity, immersion, presence, embodiment, illusion and empathy.

2 From frame to field

The main argument of VR directors concerns the “freedom” that VR technology allows against the constraints of conventional cinematography. For example, the fundamental idea of the multimedia installation (including a VR experience) Carne y Arena (2017) — one of the most debated works in the last years — is, according to its creator director Alejandro Gonzalez Iñarritu, “to experiment with VR technology in order to explore the human condition in an attempt to break the dictatorship of the frame.”¹ In fact, the screen — which is the emblem of the traditional film experience — seems to vanish in VR. The expansion of the visual horizon to 360 degrees disrupts the edges of the frame; and the viewer’s activity becomes more engaging, according to the specific type of VR and technological apparatus (from 360° videos to full immersive experiences, from Google cardboards to the most advanced head-mounted display). So what about the formal solutions that visually and narratively organize the spectator’s experience? What about audiovisual “grammar” — shot angle and size, visual composition, editing, transitions, camera movements, lighting, continuity, point of view, and other means — that construct the viewer’s perspective and guide her attention in the narrative development of the story?

Traditional cinematic techniques seem to be only partially replicable in VR. Unlike rectangular screens, a head-mounted display in fact covers a wider field of view, more similar to what we have in reality, that is, with many elements in the peripheral portion of the visual field. The screen turns from a rectangle into a sphere, and while in traditional cinematic viewing the screen limits the peripheral area, in VR the relationship between on-screen and off-screen is dynamic and more dependent on the observer’s behaviour: it changes as our head or body changes its orientation. In this sense, the “point of interest,” which is usually focused in the centre of the visual field (the so-called “fixation point”) may be located even outside the temporary peripheral area, creating a possible loss of information for the user. This has an impact on the way in which the story attracts, predicts, and directs the user’s attention, given that the latter may risk to ignore the salient visual elements or narrative events since he is attracted by a “navigational” form of exploration of the contents, instead of being guided through it. Consequently, when a story is told in VR, the user’s freedom may turn into disorientation.

These “limitations”, however, can offer creative tools to VR creators. VR directors are more and more aware that images and stories have to be constructed through a strategic use of linguistic elements such as sound, eyeline match cuts, or graphic elements, with the aim of directing the user’s attention towards the desired area, according to a new kind of editing that adopts solutions more functional to the specificity of the medium (see Bodini 2017; Dolley 2017; Mateer 2017; Bucher 2018; Williams, Love & Love 2021). In other words, the seeming narrative disaffordances of VR may turn into advantages. The temporary out-of-sight area (technically called the “zone of curiosity”), for example, can be used in the horror genre to place an unexpected presence behind the user and to surprise or terrorize him.

More generally, as Blandine Joret argues in her contribution, the user’s experience takes place in a sort of forensic framework that requires both a witnessing and an investigative activity in which he is involved in a progressive discover of evidence. Joret also helps us to understand that VR is not as much of a “post-cinema” as it seems, “at least if one is willing to acknowledge that there has always been an off-screen, exilic and embodied nature to film studies,” as theorists such as Rudolf Arnheim, André Bazin, and Dudley Andrew demonstrate.

A converse perspective is offered by Christopher Maraffi, who discusses the capacity of some room-scale VR experiences to provide the illusion of a believable world build with the same principles of the magic theatre typical of the early cinema age. This interface produces the “twin effects of simulation and dissimulation through misdirection,” that is, directing the spectator’s attention far from the mechanics of the trick.

As these and other articles, the tensions between the direction and misdirection of attention, simulation and dissimulation, awareness and unawareness of mediation, transparency and opacity of technology are very specific of VR and of its expansion from the frame (as a stable support of images and story) to a broader and changeable field that calls the user to a new kind of spectatorial agency which includes her activity in the real-time construction of visuals and development of narration.

3 From screen to scape

The passage from frame to field corresponds to the turn from the idea of screen as a surface of representation to that of the field as an environment that surrounds the user and within which the latter can actively dwell. It was in fact one of the originators of VR, Jaron Lanier (2017), who pointed in a precise direction, casting VR as a technology capable of creating alternate inhabitable worlds with their own specific expanded sensoriality. This seems to its true specificity, that which defines the ontology of this technology: “to live in the environment,” an experience that can then take on different forms (360^o VR, interactive VR, varying in the different “degrees of freedom” and movement in space that the apparatus allows).

Virtual scenarios lose their iconic and representational nature and become “an-icons” (Pinotti 2017) or “mediascapes” (Casetti 2019), that is, experiential environments that host environmental experience. This is one of most critical aspects of VR as a new storytelling medium: the irresistible tendency to offer explorable worlds as they were “environmental stories,” without a narrative organization of content — or, in other words, “non-intentional” narrations lacking in a clear or canonical teleology.

Notwithstanding — or maybe thanks to — this “navigational” mode of narrative experience, VR stories are not pre-determined but rather seem to emerge directly and exclusively from the reciprocal interactivity of the agent and the possibilities offered by the environment. In this sense, the narrative emerges from the user’s behaviour and her confrontation with the potentialities and the constraints of the spatiotemporal dimensions of narration.

Adopting an enactive approach, Elisabetta Modena and Francesco Parisi analyse a series of walking simulators and emphasize both the spatialization of narration and the actual motoric nature of the user’s action (and not merely the sensorimotor activity typical of conventional film experiences). Their analysis shows that VR allows “a creative and transformative process where both agents and audiovisual products change and modify each other” and describe this process as “zone of becoming.” Similarly, Sofia Pirandello analyses augmented reality and mixed reality “walking artworks” installed in urban spaces. These installations involve the visitors/pedestrians in a creative performance that completes the artwork and reconfigures the meaning of places.

The analyses of VR games and AR installations proposed in these articles also show that narration and the user’s bodily performance are deeply connected. As Szylvia Ruszev argues in her contribution, VR can be described as an embodied, spatial and interactive experience that is focused on the user’s body as the very origins of storyfication: “The body, through the senses of kinesthesia and proprioception, becomes a sensitive storytelling device.” Accordingly, although developed in film studies, theories and methodologies on the dynamics of spectatorial attention (cognitive psychology), on the embodied nature of simulation (social neuroscience), and on the tactile aesthetics of audiovisual experience (phenomenological philosophy) find in VR a terrain of radical relevance and application.

As many of the essays underline, literal embodiment of technology is possible with the prosthetic integration of VR devices and the human body. The head-mounted display encapsulated in the VR headset brings the image directly to the user’s eyes and forms a sort of camera obscura with her head. The more we extend our body and our mind in the virtual world, the more technology is physically close or even incorporated into our body. The new composite dispositive that arises from such a “syncretic unity” between technology and the body makes clear that VR is an elective terrain for reflection on the nature of contemporary media experience.

4 From absence to presence

Further tensions arise here between immersion and emersion, presence and absence, flow and fragment, embodiment and disembodiment, as sites of reflection on the nature of VR experience. Many articles deal with different notions of immersion and presence that the debate on the psychological characteristics of VR usually takes up. The various accounts of immersion and presence provided by a number of scholars in different fields — from philosophy to psychology, from ludology to narratology, from cognitive science to computer science (see for example Hillis 1999; Ryan 2001; Slater 2009; Noë 2012; Popat 2016; Raz 2019) — converge in their need to provide conceptual tools to explain VR’s specific capacity to situate the user in a vivid spatial and social system that is alternative to the real world. Among these theoretical attempts, Gordon Calleja’s notion of “incorporation” — elaborated in the field of game studies — seems to offer the most comprehensive and precise description of the phenomenon: “the player incorporates (in the sense of internalizing or assimilating) the game environment into consciousness while simultaneously being incorporated through the avatar into that environment” (Calleja 2011: 169). The pre-designed nature of the experience (and the scripted nature of the storyline) and the player’s cognitive activity (and the emergent stories) are coupled in a narrative tactic that characterizes ludic strategies of involvement. Adopting Calleja’s account of incorporation, Giancarlo Grossi argues in his comparison between installation art environments and dreaming, that “in virtual worlds too the story arises from the negotiation between pre-designed narrative elements and the free associations and paths created and followed by the user.” For this reason, Grossi prefers to use Céline Tricart’s (the creator of acclaimed VR experience The Key [2019]) concept of “story-living” instead of storytelling to emphasize the user’s personal experience and emotions in their interaction with — and thus contribution to creating — the narrative.

A critical point, however, concerns the nature of the user’s active participation promoted in the most advanced forms of VR. Federico Biggio’s essay reflects on the nature of interactivity (traditionally considered as the opposite of both narrative and immersion) relying on Pietro Montani’s (2014) discussion of “anaesthetization of sensibility” as the negative aspect of media immersion and the user’s relative passivity, i.e. the lack of critical distance due to subjection to the subtle influence of technology. Against the “apocalyptic” account and the phobia of “embodied media”, Emilio Garroni’s notion of “meta-operativity” and Montani’s “interactive imagination” helps Biggio to rethink VR and AR as occasions for the user to develop a creative attitude and “technical empowerment” through new forms of narrative.

VR games are exemplary of these dynamics for their ergodicity, that is, an extra-noematic effort that goes beyond the mere decoding or interpretation of narration. In his article Nicolas Bilchi offers a paradigmatic example of a VR game in which the illusionistic mimetic potential of VR is confronted with its tendency to bring “consciousness about the impalpable and disembodied essence of the virtual body during the (simulated) physical interaction with the environment”. This “disillusion” or “break of presence” contributes to the inadequate sense of body ownership that the VR apparatus is unable to overcome. In this sense, Bilchi concludes that rather than emphasizing its immersion potential, VR should be thought of and experimented with as a tool to for fostering critical and reflexive attitudes.

Indeed, this attitude is one of the specificities of VR art projects. The world of the arts plays a fundamental role due to its ability to experiment with languages and cognitive models. From a more overtly dreamlike (if not psychedelic) direction (that is, free from causal relations and an explicitly realistic dimension) to an intrinsic relation with performativity, the paths proposed by the world of art (which among other things is free of the hindrances of film and television production and can base itself in a network of galleries and museum initiatives that already have familiarity with technology, at least through video art) offer concrete proposals and above all a high level of aesthetic experimentation.

As Valentino Catricalà and Francesco Spampinato highlight in their article, “contemporary artists are less interested in exploring VR’s illusionistic and immersive dynamics than deconstructing the VR medium itself.” In the VR artworks they analyse, the immersive and multi-sensory experiences offered by VR are not emphasized for their own sake, but rather as means that “make users aware of the alienating and desensitizing impact of digital technologies.”

In short, to be in a three-dimensional environment, to move within it, explore it, and see it change is already in itself an amplification that definitively transforms the spectator (etymologically, the one who sits before the spectacle) into an agent. This introduces the issue of interactivity and its capacity to pose the question of the usage of a new narrative dimension. Interactive storytelling finds in immersive storytelling a point of intersection that heralds a wide range of possibilities. This opens up a vast field for observation and analysis, which entails examining not only CGI and sensors, but also the acquisition of data and how they enable in real time, with the support of AI, machine and deep learning and generative algorithms. The technological question thus becomes central, not only and not so much in terms of its functioning, but rather as a way to investigate the dynamics and modalities in which the functions are enabled and impacts are generated (and consequently, form and aesthetics), but also to interpret the sociocultural context in which all this occurs, identifying a post-media horizon that would be better defined as “post-digital”. Raffaele Pavoni’s article on spatial sound in VR and in cinema — approached with theoretical tools borrowed from both sound studies and software studies — follows this direction.

5 From identification to empathy

As most of the case studies chosen by the authors demonstrate, the cinematic “genre” from which VR takes the most is that of documentary. Whether it be naturalistic, scientific, journalistic, poetic, social, or auteurist, non-fiction narration seems to be the site at which the potential of immersive language bursts forth: to be directly in the environment that is being visited, to be in the real situation that is being dealt with.

The mainstream rhetoric in contemporary discourse promotes the idea of VR as “the ultimate empathy machine” (Milk 2017) for its capacity to create a vivid impression of being with others (even if the latter are images or avatars) and thus offer an effective form of compassionate understanding. It should be noted that, if empathy is a way to deeply understand the condition of the other, paradoxically, when technologically mediated, it can also be the most obvious way to keep at a safe distance from tragedies that we would never want to experience first-hand. As Neta Alexander explains in her essay on VR documentaries on the Shoah, the illusion of co-presence or co-witnessing does not guarantee automatically an empathetic response or a deeper understanding. VR risks putting the user at an “improper distance” (too close) to a sensitive topic and to conflate attention and learning with “slacktivism” (the support a social cause through social media or online petitions, with very little commitment) or “identity tourism” (putting oneself in the other’s shoes as a leisure activity) and thus fetishizes the other. As Paul Bloom (2016) argues, empathy generates pleasure for its ability to make us feel involved with others, but it is far from a valid moral and decision-making guide: “VR doesn’t actually help you appreciate what it’s like to be a refugee […] In fact, it can be dangerously misleading” (2017).

Along the same lines, Dirk Eitzen adopts a cognitivist account of empathy and clarifies that a “moral ambivalence” arises in the case of VR documentaries on migration or similar topics. Given that we can imagine being in the other’s shoes, the strong and literal subjectification of experience promoted by VR (the shift from third and second person to first person) is not the best way to genuinely share the experience of the other: “We may think we are imagining somebody else’s experience but […] our self-absorption tends to eclipse genuine understanding.” Paradoxically, Eitzen concludes, conventional (documentary) cinema and third-person storytelling remains the most effective medium to open a window into the life, thoughts and feelings of somebody else.

In regard to the subjectification of the point of view in VR storytelling, Joseph Fischer’s analysis of theme parks as narrative spaces is based on the idea that diegetic worlds “actively involve the spectator and position them as if they are the frame of a camera exploring and creating their own stories within the multi-acre diegesis.”

6 Extended VR

Even in the light of the interesting questions and significant case studies that will emerge from the reading of the various contributions, we are today faced with an indisputable fact: despite the predictions made some years ago, from both the production and technological sectors, VR apparatuses have not broken into the market; there has been no diffusion of VR arcades, let alone a market that would produce and distribute content. VR as an entertainment medium is not experiencing success (not even in the more vibrant field of gaming). On the other hand, VR represents a fundamental component in the field of “extended” immersive technologies (so-called XR). XR - and thus VR - play a primary role in the creation of next-generation immersive and interactive environments that have been developed in various industrial sectors (from the automobile industry to aeronautics), in simulation contexts (military usages as well as medical, surgical, and scientific ones), in a museal and didactic context (with immersive classrooms or galleries), in the implementation of architectural and urbanist projects, and finally in communications, with the development of immersive platforms and social media.

It is this “extension” of VR language that seems to offer the most interesting suggestions as to the possible narrative usages of the medium. Every VR experience is, in its way, experimental (hence the fundamental importance of the realm of arts inclined to work creatively on their languages), thus a little laboratory in itself for discoveries and practical applications, even for narrative works.

We are probably still at the beginning of a path of the affirmation and refinement of a medium that, if it manages to survive, will have to develop a greater awareness of its own means and articulate a new language.