1 Introduction

One of the most common experiences nowadays is certainly that which takes place in front of a screen. Be it a flat, framed screen, or a surrounding, simulated environment, we are constantly exposed to a multifaceted galaxy of audiovisual experiences: movies, video games, TV shows, social media, public installations, and so forth.

In this article, we will focus on video games and VR as particular cases of storytelling. In doing so, in section 2 we introduce and discuss three terms often adopted to describe video games and VR experiences, namely flow, immersion, and presence. In section 3, we try to make the same analysis but from a neuropsychological perspective and argue that both speculative and experimental studies fail to isolate these phenomena as if they were sharply separated in real media experiences. In our opinion, they fail because of the theoretical assumptions which usually frame the problem and also because of the extremely vast and heterogeneous supply of screen-mediated experiences.

In section 4, we start to illustrate such heterogeneity, particularly by presenting “walking simulators,” audiovisual products which, even though designed and sold as video games, lack crucial elements traditionally found in videogames and which introduce storytelling experiences based on solitary explorations.

The importance of the connection between stories and environments is discussed in section 5 as a fundamental nexus in the creation of immersive experiences. By further exploring the concept of experience, we argue that those experiences which occur in image-worlds, such as video games and VR, reveal a “zone of becoming,” which is a creative and transformative process where both agents and audiovisual products change and modify each other.

Finally, in section 6, we continue to develop this transformative view of experience-as-becoming by relating flow, presence, and immersion as cognitive outcomes disciplined by opportunities and constraints.

2 Flow, Immersion, and Presence

Let us start by establishing the main, irreducible difference between generic audiovisual stories and video games. This difference has been famously discussed, among others, by Espen Aarseth (1997), precisely in order to liberate the interpretation of video games from the close links connecting it to traditional texts. In his book, the author distinguishes between ergodic and non-ergodic texts. In non-ergodic texts, the relationship between the text and its reading is trivial, in the sense that the second directly comes from the first. In ergodic texts, in contrast, the relationship becomes non-trivial, in virtue of the active intervention of the reader, who chooses the path of the story will take: “The effort and energy demanded by the cybertext of its reader raise the stakes of interpretation to those of intervention. Trying to know a cybertext is an investment of personal improvisation that can result in either intimacy or failure. The tensions at work in a cybertext, while not incompatible with those of narrative desire, are also something more: a struggle not merely for interpretative insight but also for narrative control” (Aarseth 1997: 4).

Nevertheless, beyond this fundamental difference, many other aspects are shared by ergodic and non-ergodic texts. In particular, three terms seem to capture the multidimensional way audiovisual media engage users: flow, immersion, and presence.

The first term is flow. Sometimes used interchangeably with immersion (Michailidis et al. 2018), but certainly less ambiguous, flow was originally introduced by the psychologist Mihály Csíkszentmihályi (1975); it indicates a state of consciousness characterized by certain specific conditions: a challenging activity that requires skills; the merging of action and awareness; clear goals and feedback; concentration on the task at hand; the paradox of control; the loss of self-consciousness; the transformation of time; the autotelic experience (Csíkszentmihályi 1990). Some of the above conditions can also be attributed to immersion and presence, but others are exclusively specific to flow. One especially seems to be possible only for ergodic texts: clear goals and feedback. Admittedly, if goals can be pursued in non-ergodic texts, it does not make any sense to include feedback too. Only by acting, feedback can be expected. Flow is then the condition that seems highly specific for activities that engage the “reader” performatively.

The most problematic term is immersion, for two reasons: it is semantically vast and applicable to both ergodic and non-ergodic media (see Calleja 2011 for a discussion). In fact, if we assume the definition offered by Janet Murray, which is considered one of the most authoritative in game studies, immersion sounds like this: “Immersion is a metaphorical term derived from the physical experience of being submerged in water. We seek the same feeling from a psychologically immersive experience that we do from a plunge in the ocean or swimming pool: the sensation of being surrounded by a completely other reality, as different as water is from air, that takes over all of our attention, our whole perceptual apparatus” (Murray 2017: 99). Besides, Murray adds, immersion is a “participatory activity” that requires active involvement and complete attention.

Presented in this way, immersion seems to be something encompassing both physical and psychological dimensions. Being immersed in a story echoes the perceptual, environmental sensation of being physically immersed in it. Simultaneously, as Murray explicitly suggests, immersion is increased not by the mere suspension of disbelief, as S.T. Coleridge remarkably pointed out, but is empowered by the creation of beliefs concerning fictional world. Immersion is, therefore, an all-inclusive and vague concept, since it involves both physical and psychological aspects (McMahan 2003).

A way to reduce this vagueness consists in stressing the very physicality of immersion. A term that might fit this purpose and has an honourable tradition in game studies and computer sciences is presence. Derived from Marvin Minsky’s concept of telepresence, strictly speaking this term applies to the physical, and ego-spatially referred sensation of being somewhere, not metaphorically, but concretely and sensomotorically. In a certain way, presence could be interpreted as a deeper kind of immersion, where the agent experiences the physical and spatial sensation of having abandoned his/her environmental surroundings. Gordon Calleja defined “immersion as absorption” and “immersion as transportation” respectively as the psychological condition of being enchanted by a story and as the physical projection into the mediated world (Calleja 2011: 26-7).

It is important to notice that these terms are frequently used interchangeably, often generating confusion and misunderstanding. As we will see, ambiguity and polysemy characterize their use. Relying on experimental evidence and psychological and philosophical speculations, we will try to offer a simplified picture of the problem.

3 Neurocognitive Data and Psychological Theories

There are mainly two ways through which video gaming can be studied: on the one hand, by checking the cognitive effects of this activity on human cognition (Green et al. 2003, Bavelier et al. 2011, Gozli et al. 2014, Boot 2015, Unsworth et al. 2015, Chabris 2017, Bediou et al. 2018, Dale et al. 2020), on the other hand, by trying to understand what characterizes the experience of game playing. Even though both ways are important for a full understanding of the phenomenon investigated, the aim of this paper is to address the second.

Being in the flow is a multifaceted experience difficult even to self-report. It is very hard, then, to model an experimental setting and to choose what to investigate and what to exclude (see Weber et al. 2009, and Khoshnoud et al. 2020 for a discussion of methodological issues). Besides, not every game is the same, so game variety can complicate things further. With this awareness in mind, the first problem consists in circumscribing the starting hypothesis. The study by Arne Dietrich (2004), for example, refers to the implicit/explicit dichotomy to differentiate between possible cognitive strategies: the explicit system is associated with the higher and temporally extended cognitive functions (flexibility), while the implicit system is associated with skill-based, and faster cognitive reactions (efficiency). By relying on this distinction, the author suggests that “optimal performance involving a real-time sensorimotor integration task is associated with maximal implicitness of the task’s execution. […] it follows from this proposal that a flow experience must occur during a state of transient hypofrontality that can bring about the inhibition of the explicit system.” (Dietrich 2004: 757).

The term “hypofrontality” is relevant here. It echoes a lack of long-term planning, as well as reduced monitoring of conscious activity. Thus, it seems that you need to lose yourself to get in the flow. More precisely, as Csíkszentmihályi points out, you need to lose the “consciousness of the self” (1990: 64), meaning that you have to ignore the conscious representation of who you are, but not the immediate sensorimotor feeling of being you. Employing the terminology of embodied cognition, you have to forget your body image, but not your body schema, and neither do you have to “lose your mind,” as athletes and performers know perfectly well (Francesconi and Gallagher 2019, Gallagher 2018).

Active sensorimotor involvement seems to be a crucial component for getting into the flow. To test the degree and the extension of this involvement, Martin Klasen and colleagues tried to measure through fMRI brain network activations while users were playing specific game contents (a first-person shot fight game). This method assumes that a state of flow can be inferred by the game situation the user is involved in (low, medium, and high focus) and the success of the game session. A conjunction analysis confronting brain activations showed the emergence of a motor simulation network responsible for the insurgence of flow. The authors point out that “this sensorimotor activation reflects the simulation of physical activity in the game and that activation of this ‘simulation network’ contributes to the emergence of flow.” (Klasen et al. 2012: 490)

The problem with the above experiment is that it relies on a specific game, or better on the experience caused by the game. To offer a more general and comprehensive description of flow, René Weber and colleagues shifted the core of the analysis from the content of the game to the working condition of the brain. Their definition runs as follows: “In the media context, flow is a discrete, energetically optimized, and gratifying experience resulting from the synchronization of attentional and reward networks under condition of balance between challenge and skill.” (Weber et al. 2009: 412). The aim then was to describe the flow independently from the kind of content experienced by the user. In this case, though, even if the overall description of flow offered is more general and content-independent, nonetheless it can be strictly circumscribed to a given psychological domain. This is why the overlap between flow and other phenomena, mainly immersion, is a constant worry in every experimental setting.

A preliminary study conducted by Emily Brown and Paul Cairns (2004), for example, shows how slippery and hard to define the term immersion is. By using questionnaires administered to gamers, they found that immersion can be threefold: “engagement” is the first phase, corresponding to the willingness in terms of time and personal preference of the gamer to play a given game; after that, the gamer experiences “engrossment” if he/she is emotionally involved; finally, the gamer can experience “total immersion” if he/she has the illusion of being cut off from the surrounding space.

In another experiment designed to combine objective measures and subjective reports, Charlene Jennet and colleagues (2008) started separating immersion from flow and cognitive absorption. Although they conducted three experiments and developed a questionnaire designed explicitly to investigate immersion, unexpected results and a great degree of undefined variables at many levels (conceptual, practical, and instrumental), forced them to conclude that much work still needs to be done. A recent attempt to unify experimental evidence and conceptual framework seems to confirm the trend of this “conceptual challenge.” In their study, Lazaros Michaidilis and colleagues (2018) conclude that immersion and flow should not be considered separate processes, at least for the moment, considering the degree of generalizability currently characterizing the debate. To justify a sharp separation between these concepts, they claim, we need more neuropsychological and experimental evidence.

As a further confirmation of the mutual permeability of the terms investigated so far, David Weibel and Bartholomäus Wissmath assert that “flow could thus be described as immersion into an activity (i.e., the gaming action), whereas presence refers to a sense of spatial immersion in a mediated world.” (Weibel and Wissmath 2009: 3, italics added). Despite their relaxed use of the terms, we agree with the authors in considering flow as dependent on the agent’s activity and presence on the kind of technological artefact employed. As the term clearly suggests, in fact, the feeling of presence occurs when the agent experiences a spatial and sensorimotor sensation of being in a place. The sensation is clear to all of us because it defines our daily life, but it can be simulated by the deployment of technical artefacts if the media engagements produced satisfy specific conditions.

It is no accident, then, that the concept of presence has been investigated in virtual reality (VR), the only technological equipment that can allow a sensorimotor alienation/distancing away from the physical environment towards a digital one. Mel Slater and Sylvia Wilbur (1997) proposed a model that has been influencing the debate enormously. Once again, immersion and presence are presented as similar psychological states. In their paper, they think of presence as “an increasing function of immersion in all its aspects” (Slater and Wilbur 1997: 606). But what creates the concrete shift from immersion to presence is the reconstructed physicality of both the virtual environment and of the body, which is transporting experience from one place to another, and not just/simply metaphorically. A virtual body has to enact a virtual environment: the more lawful (in a Gibsonian sense) virtual reality responds to the agent’s intentions, the higher the feeling of presence experienced (Sanchez-Vives and Slater 2005, see Slater and Sanchez-Vives 2016 for an updated discussion).

Flow, immersion, and presence are thus three interconnected and mutually permeable concepts referring to a vast array of psychological and behavioural states. Both theoretical and experimental research seem incapable of isolating such terms unequivocally, and the approaches that try to do that fail, in our opinion, in the enterprise. As we will see, this failure is motivated by the heterogeneity and hybridization of interactive experiences available and the theoretical assumptions which frame the problem. We are going to discuss them both.

4 Immersive Stories and Walks

In recent years, the video game market has seen the birth and rise of products created by individuals or smaller development teams that have put immersive space exploration at the centre of their projects. More precisely, many of these video games have transformed the exploration of space into a game mechanic, indeed into the only mechanic possible within a three-dimensional space. Strictly under a mode of classification of video games based on mechanics — at least traditionally, although not exclusively (for a debate on video game genres see: Crawford 1984; Myers 1999; Bittanti 1999; Apperley 2006; Arsenault 2011) — these games have been called in a derogatory way “walking simulators.” Therefore, the focus is placed on interaction methods rather than on the other elements which characterize these “games” (Montembeault and Deslongchamps-Gagnon 2019). Additionally, as we will see, most of these video games enhance the role of storytelling and “environmental storytelling” (Jenkins 2004) in order to increase the flow. In this respect, these are challenging examples to add some remarks to the topics we introduced above.

In Gone Home (Fullbright 2013) for example, I search “my” empty house looking for my family. Room by room, objects and short texts help me understand what happened to my sister while I was away, where she went and why. I take pleasure in unravelling the story from a first-person perspective. I feel immersed in this abandoned house, and I perceive its deep sadness.

Dear Esther (2012) and Everybody’s Gone to the Rapture (2015) are productions of the English game development studio The Chinese Room. The first is set on an island in the Hebrides and consists of an exploratory “immersion” in the place, which corresponds to the progressive unravelling of an apparently disconnected story read by a man’s voice. An enveloping soundtrack makes this journey even more immersive in the approximately two hours of play. Everybody’s Gone to the Rapture is once again based on exploration and the mystery surrounding a deserted town in which the environment itself becomes the bearer of memories and clues.

In Proteus (Key and Kanaga 2013), the exploration takes place on a pixelated island. The seasons follow one another and the nature of the randomly generated environment changes shape. It is an almost abstract experience, in which there is no real story apart from the narratives created by our immersion in space and by the objects and events we perceive.

Other games from the same period are occasionally considered walking simulators, for example Journey (Thatgamecompany 2012). In this award-winning game, “Alone and surrounded by miles of burning, sprawling desert, you soon discover the looming mountaintop is your goal. The passage will not be easy, but this experience of a lifetime will help you discover who you are, what this place is, as you arrive at your purpose” (Journey 2012).

David O’Reilly’s Everything (Double Fine Productions 2017) presents the idea of exploration with an assortment of possible points of view thanks to the use of different avatars. The player takes the form and point of view of different animals, plants, minerals, microorganisms, and more in order to experience different interactions with the environment, from the microscopic to the macroscopic.

Exploration of the environment is not a prerogative of these games. More and more often even large Triple AAA productions have opted for forms of ever-greater freedom of movement in the typical form of the sandbox and open world games, from the GTA (Grand Theft Auto) and Assassin’s Creed sagas to Death Stranding (2020). However, what characterizes this specific type of game, the “walking simulator,” is placing at the centre of the experience immersion within a particular environment (also recalling the distinction between mechanical and narrative immersion suggested by Mason 2013).

Nevertheless, this new type of video game has put the very definition of “video game” into serious question, because of its lack of goals and challenges. Game Studies scholar Jesper Juul recently addressed this topic in an article in which he analyzes these games. Juul points out the absence of challenges and gameplay. On this, he builds an idea based on different aesthetic layers in which the third level (“Aesthetics III: Aesthetics of the aesthetics of the aesthetics of video games”) would correspond to “walking simulators” such as the already mentioned Gone Home, Proteus, Dear Esther and The Path (Tale of Tales 2009) or Firewatch (Campo Santo 2016).

In a radical departure in game history, walking simulators, therefore, reject the gameplay and strategy optimization which characterize most games, but in doing so, walking simulators present a quite conservative and traditional idea of aesthetics, in order to create video games that can fit in art gallery settings (Juul 2018).

Therefore, Juul notes the rejection of gameplay strategies as the main element of these experiences. As evident from our previous description, this was not the first characteristic we noted: our first reaction was not related to the absence or the rejection of strategy optimization, as Juul suggested.

Our impression was the one we described above: immersion within a space. It is no coincidence that in these video games, the interface is minimal or even absent while it is perceived as an element that structures and masks the relationship with a world which is instead intended to be as fluid as possible. Therefore, in most cases, the user is catapulted without information into an environment in which she does not find precise indications or specific goals.

Does it make sense to ask whether or not it is a video game? From Juul’s perspective, it certainly does, considering his position in the debate about video games and narration and the importance he assigned to interaction (Juul 2001).

In his article significantly titled Games telling stories?, the scholar entered the debate between “narratologists” and “ludologists,” which can be ascribed to the origin of a new discipline dedicated to studying video games. One of the most debated cases was that of Tetris (Aleksej Leonidovič Pažitnov 1984). If video games tell stories, what story could Tetris ever tell, a game in which there appeared none of the minimal elements that give rise to a narration in the traditional sense? Some said that even Tetris had a story (Murray 2017), and some claimed that it did not have it at all (Eskelinen 2001). Today, twenty years later, the question of whether a video game tells a story appears almost rhetorical, considering the development of narration through the video game medium. Still, at the time, this question was useful in identifying specific tools to study this interactive medium.

However, this debate may suggest a question about the role of narrative in these games. We could provocatively put these video game experiences at opposite poles: on the one hand only gameplay and no story (Tetris); on the other hand, minimal game mechanics and only narrative (Gone Home, Dear Esther, Everybody’s Gone to the Rapture) or the so-called “environmental storytelling” (Jenkins 2004) as in Journey, Proteus, Everything.

However, no one questioned that Tetris was a video game, while the very nature of Gone Home or Proteus’s has been questioned. According to Juul, this “rejection of the gameplay” would, therefore, involve an attempt to present these games as artistic forms of an aesthetic nature in the most conservative sense of the term.

5 Immersive Experiences

Actually, what Juul claims could be proved correct by the fact that recently many artists have created similar experiences by designing immersive artworks, with the sole purpose of putting the user at the centre of an environment and some form of narration.

We are thinking in particular of works created using Virtual Reality technology such as Endodrome (2019) by Dominique Gonzalez-Foerster, Chalkroom (2017) and To the Moon (2018) by Laurie Anderson & Hsin-Chien Huang, or It Will End in Stars (2018) by Nathalie Djurberg and Hans Berg.

Endodrome, presented at the latest Venice Biennale in 2019 by French artist Gonzalez-Foerster “immerses audiences in an abstract and meditative environment” (Endodrome 2019). Chalkroom is “a virtual reality work by Laurie Anderson and Hsin-Chien Huang in which the reader flies through an enormous structure made of words, drawings and stories. Once you enter you are free to roam and fly. Words sail through the air as emails. They fall into dust. They form and reform” (Chalkroom – Laurie Anderson 2018). In To the Moon, a 15-minute VR experience, the viewer is shot out from earth to the surface of the moon on which she can walk and interact. According to the Director’s statement published on the Venice Biennale website, “To The Moon, allows the viewer to literally walk into a work of art” (To the Moon | Biennale Cinema 2019).

Finally, Djurberg and Berg VR experience is a modern version of a fable with the big bad wolf:

Viewers begin their journey deep in a forest, entering a clearing to discover a wooden cabin. Inside, a wolf sits in front of a glowing fire — the building’s chintz interior undercut by a creeping sense of unease. Hand-written text, which appears hanging in mid-air, conveys the wolf’s thoughts, merging digital technology with an aesthetic closer to silent cinema. He describes the virtual environment as the “shadowside,” delivering a confession that merges themes common in Djurberg’s practice; fear, guilt and animalistic impuls (Acute Art | Nathalie Djurberg & Hans Berg 2018).

We have not mentioned these VR productions by chance although we could have cited other examples of experimental three-dimensional immersive environments made by other artists such as The Night Journey (2007-2018) by Bill Viola. In fact, we believe that this new technology and the debate that has arisen around it, help shed light on an increasingly widespread phenomenon. Indeed, the design of these new products must be linked to a broader field that sees the growing demand for increasingly immersive forms of narratives in all areas, especially in entertainment, and beyond (for example in non-fictional products and art). Each experience promises to be immersive, to envelop the viewer and make her feel involved. In fact, Fabienne Liptay and Burcu Dogramaci — in the introduction to the book edited by them, with the title Immersion in the Visual Art and Media — note that:

the term immersion (derived from Latin “immersio”) may refer to any act or experience of plunging into something, without necessarily applying to a computer-generated virtual environment. Due to the term’s wide variety of uses, especially in the English language, a baptismal font or a swimming pool, a chemical solution or a medicinal bath, the shadow of a planet or a foreign language can equally serve as immersive “media” (Liptay and Dogramaci 2016: 1).

The video game sector is no exception, and this goal has been pursued — and in part achieved well before others — through two main objectives: the creation of coherent curated environments and narrations with which to interact. As far as consistency is concerned, this is an even more important quality than photorealism. As abstract as it is, a game world works if it is credible as a place where the narratives occur. The same goes for the narrative, more or less present and detailed; it must necessarily be coherent to the world so as not to break the contract with the user.

But the point we would like to focus on when describing these products, which are so different and created in apparently distant contexts, is that there is a leitmotif in the way we describe them. As we saw above, more and more often we speak, in fact, of “immersive experiences.” But what is an “experience”? And why does an experience have to be immersive?

Still, today, thinking about the media, we never describe with this word the act of reading a novel or watching a film. However, we continuously do so when we refer to a video game, an installation or a virtual reality environment. Indeed, as Ruggero Eugeni has pointed out, the concept of experience is a crucial concept of our contemporaneity. Eugeni describes the experience starting from its double dictionary definition as a patrimony of acquired knowledge, and as “the course of the events of consciousness that take place starting from the concrete and living placement of the subject within a world” (Eugeni 2010: 25, our translation). Being of an interior and personal nature, therefore, every experience requires an interpretative action, a “hypothetical process.” This aspect reflects the etymological meaning of “experiment” (the verb experiri means to search through tests and attempts), and so the idea of ​ something involving repetition and effort as in a video game.

Experience is based on interpretation. It is dynamic and made of three layers: after a survey of the sensitive data of what is present around her, the subject proceeds to a narrative ordering of what she perceives by comparing it with the experiences of other people (relational harmony). Therefore, considerable importance is assigned to the “design of the experience,” or rather to the construction of the experience itself, the real goal of the semiotic approach. Indeed, from a semiotic and media perspective, Eugeni describes the video game as an “environmental experience,” in contrast to the closed experience of the text. From the point of view of the relationship with the everyday world, however, it is difficult to catalogue, since it can result both in continuity and in discontinuity with this world: it can therefore be “factual” as much as “fictional,” “participatory” and “aesthetic” (Eugeni 2010: 293). A few years later in the book La condizione postmediale, Eugeni (2015) tackles a crucial issue for us, namely that of the first person shot as an emblematic post-media figure as the interpreter of a subjectivisation of experience.

Thus, it seems unsurprising to read how products created with the purpose of placing the user inside an environment are defined as “immersive narrative experiences” such as what happens in VR where you enter an image-world (Pinotti 2020). And the same happens for these “walking simulators.” Therefore, for instance, Gone Home puts you within “An Immersive Place: Return to the 1990s by visiting a home where every detail has been carefully recreated, and the sounds of a rainstorm outside wrap you in the experience” (Gone Home on Steam 2013) and Dear Esther is “a first-person game about love, loss, guilt and redemption. Driven by story and immersion rather than traditional mechanics, it’s an uncompromisingly emotional experience” (Dear Esther – The Chinese Room 2012); Proteus “is a game about exploration and immersion in a dream-like island world where the soundtrack to your play is created by your surroundings. Played in first-person, the primary means of interaction is simply your presence in the world and how you observe it” (Proteus on Steam 2012).

However, it is surprising to see how even games based on mechanics are now offered as emotional experiences. Tetris in its reinterpretation by Tetsuya Mizuguchi entitled Tetris Effect and also available in VR “amplifies this magical feeling of total immersion by surrounding you with fantastic, fully three-dimensional worlds that react and evolve based on how you play” (TETRIS® EFFECT 2018).

The fact that a series of events happen in an environment where we (or our avatar) can move and that their meaning is dynamically constructed through our interaction, transforms these new forms of storytelling into experiences. Precisely for this reason, we believe that in a narratological language it will be increasingly important to consider the concept of “experience” as a new tool not only for ekphrasis (from Ancient Greek ἐκ “outside” and ϕράζω “to speak;” to designate an inanimate object with a name) — as already happens — but above all in terms of design and analysis.

In this respect, analysing the concept of agency in video games and its relationship with immersion, Souvik Mukherjee proposes a new definition of how the player relates to the text which seems very close to the concept of the experience. For him, “the process of choice-creation and the actualisation of possibilities (in the action-image) is not explainable as agency but is more aptly described as a ‘becoming’”. Moreover: “Instead of the carefully predetermined agency of conscious human thought, the action in the computer game develops as a result of the process of ‘becoming’ in the interaction of the ‘thought’ of both the human and the computer. Therefore, ‘becoming’ not only explains the development of the action in the computer game; simultaneously, the development of the subject is described” (Mukherjee 2015: 167-168).

Experience design as the design of this “zone of becoming” (Mukherjee 2015: 168), therefore, constitutes an articulated approach to a multimedia product that aims to be immersive, both for video games and VR.

6 Agency as Immersive Becoming: Insights from 4E Cognition

What we have said so far can be summarized as follows: Being in the flow is a condition which is task-dependent, and necessarily produced by active sensorimotor involvement; immersion is a broad, all-encompassing phenomenon that can be achieved in different (all?) media contexts, whilst presence is a kind of immersion that is environmentally and ego-spatially characterized. From the perspective of the product, flow requires feedback and then is achievable during video gaming; presence is obtained only if a proper environmental simulation is capable of creating the illusion of being physically elsewhere (VR); immersion can be experienced in different audiovisual products (ergodic and non-ergodic texts), as well as through different technological supports (books, movies, video games, VR).

Even though our proposal maintains the terminological distinction between the three terms discussed, it questions the possibility of their being discreetly separated and investigated as “pure” cognitive phenomena.Importantly, this does not mean that they are medium-specific either, or dependent on the bare materiality of the medium. They depend on the relation between the agent, the medium, and the environment. When we play video games, watch movies, wear a head-mounted display to interact, and so forth, we are not just doing something, we are becoming someone (Di Paolo 2020). For this perspective to be maintained, a different conception of cognition is needed. One that does not assume the subject as the immutable starting point of the process, but as a peer element of the whole system.

Such a conception can be found in 4E cognition (Newen et al. 2018, Menary 2010). To understand what “E” stands for, we have to start from the first of them: embodiment. Roughly speaking, “embodiment” is the label defining the epistemological predisposition to deem cognition as inextricably tied to the cognizer’s body. It is impossible, embodied cognitive scientists claim, to fully understand cognition if the body is not part of the formula. Put in other way, the point is that the explanation of cognitive phenomena cannot rely on the brain alone (Gallagher 2005, Shapiro 2019). This predisposition is now mainstream and has influenced media research too (Nannicelli 2014, Fingerhut 2021). In general, this approach has offered a clear, direct explanation, supported by experimental data philosophically investigated, of some neurocognitive mechanisms involved in the appreciation of fictional worlds. The approach has received much attention worldwide, and now is also applied to music (Schiavio et al. 2017) paintings (Freedberg and Gallese 2007), literature (Wojciehowski and Gallese 2011, Kukkonen 2019), moving pictures (Eugeni and D’Aloia 2014, Smith 2017, Gallese and Guerra 2020), and video games (Gregersen and Grodal 2009).

If embodiment has had the merit of recognizing the importance of the body, another theoretical shift acknowledged that the environment and the things included in it play a role in defining cognition. Beyond being embodied, then, cognition is extended, embedded, and enactive: Three more “Es” to highlight the prominent role that material artefacts and social institutions play in the cognitive behaviour of the agent, who was traditionally pictured, at the beginning of the cognitive revolution in mind sciences, as being one with his/her subpersonal brain processes.

It is impossible and far beyond the space remaining even to introduce the nuances that define the differences between the Es. Instead, we prefer to draw attention to what they have in common: cognition does not occur just in the head but involves — more or less constitutively — the surrounding world (Durt et al. 2017). Even if our sensorimotor skills evolved for natural purposes, our condition as Homo faber (Ihde and Malafouris 2019) became astonishingly evident at a certain point in our history. We are organisms whose natural condition is to be retroactively shaped by the things we have made. In this scenario, audiovisual media occupy a special place, since they strongly stimulate components of this ancient sensorimotor engagement by creating alternative simulated or recorded worlds that can be explored.

We contend that if framed inside this epistemology, the terms traditionally used in game studies — flow, presence, and immersion — reveal the different ways the agent’s experience is transformed by audiovisual media. The point stressed more or less radically by 4E cognition is that there is not a pre-given agent who watches, interacts with, or explores pre-defined mediated worlds, but there are relations between audiovisual worlds and bodies which can be understood only by scrutinizing the effects they exert “metaplastically” (Malafouris 2013) on each other.

The transformative effects of this becoming occur on multiple time scales and to different degrees. They affect personal and social dimensions of life: “Human becoming is not a genetic setup or an evolutionary stage, but an open and ongoing process of creative engagement with the material world” (Malafouris 2021). If we adopt this perspective, it becomes clearer how and why specific psychological behaviours can be expected or are impossible to get; it is like a “dance” between opportunities and constraints.

For instance, flow is constrained by the need for feedback. It is not by chance that the term has had a considerable impact on video games and sports psychology. Both video gaming and sports are activities based on performing actions and adequately reacting to stimuli. In addition, they require systems of measurement and evaluation which confer value to the actions performed. Agentive involvement, constant feedback, and evaluation, are therefore key features of video games. An example of the transformative effect potentially induced by long-lasting exposure to video games, intensified by states of flow, is the so-called “Proteus Effect”: “Users who are deindividuated in online environments may adhere to a new identity that is inferred from their avatars […] they conform to the behaviour that they believe others would expect them to have” (Yee and Bailenson 2007: 274).

To elicit virtual presence, on the other hand, the mediated environment has to offer the agent the opportunity to satisfy what Kevin O’Regan and Alva Noë called “sensorimotor contingencies” (O’Regan and Noë 2011). In the context of media engagement, it occurs only when a (virtual) body interrogates a (simulated) environment: “Presence is defined as the non-mediated (prereflexive) perception of successfully transforming intentions in action (enaction) within an external world” (Riva 2009: 160). The transformative effect in this case has the character of illusion. The out-of-body experience is the phenomenological illusion that one’s body is dislocated out there in a virtual space provided by a head-mounted display (Ehrsson 2007, Lenggenhager et al. 2007, Maselli and Slater 2013, Parisi 2021). The feeling of being elsewhere is so strong that a highly stable phenomenological state such as identifying with one’s body can be disrupted.

Finally, immersion presents no absolute constraints: it may arise in many different circumstances because it does not require direct physical activity reinforced by congruent feedback. We can be immersed in our thoughts while lying on our couch. This is why we believe that immersion is a critical element of media engagement. In a certain way, it is a precondition for (or an implication of) the other two: if you feel present somewhere, this automatically implies that you feel immersed too; the same can be said when you are in the flow. Consequently, we want to emphasize the centrality of immersion and point out that the narrative aspect of any media engagement enhances immersion. We can conceive stories as expedients for entering the fictional world and for the maintenance of long-lasting, shareable experiences. A story immediately brings us into the fictional world, possibly facilitating engagement with the gameplay. It also may create an emotional bond and provide the opportunity to relive (reenact) the story, especially with other people.