Animal-oriented virtual environments: illusion, dilation, and discovery

Cognitive neuroethology as an uncanny valley

There is evidence that these factors are most relevant to animal behavior research, for which naturalistic settings are of primary importance (Zupanc, 2010). But how much of the environment must be replicated in order for an animal to recognize it as “just like the real thing”? One way this can be characterized is through the uncanny valley phenomenon. The uncanny valley characterizes the subjectivity inherent in how observers perceive and act upon virtual environment avatars and robots that embody various degrees of realism (see Figure 1). The uncanny valley is based on an emotional response occurring in the very early stages of perceptual processing, which can be elicited for any object that generates an emotional response or involves recognition mechanisms. Both emotional response and recognition result from experience, which emerges in development and occurs in non-human contexts (Lewkowicz & Ghazanfar, 2012). Experience also conditions the classification of stimuli as being real or a facsimile in terms of recognition. Whenever a real object clearly has the attributes of such, the early emotional response resolves the ambiguity of classification as real or virtual (Steckenfinger & Ghazanfar, 2009). It is when this ambiguity cannot be resolved that the problem lies.

For purposes of this paper, let us map recognition to classification using the uncanny valley curve as a referent. Initially (see Figure 1A), the more “real” an object becomes, the more it is associated with its real-world analogue. This phase of the curve is associated with gains in sensory fidelity. The second phase of the curve (see Figure 1B), which consists of two inflection points, is associated with a drop-off in the feeling of realness just before a fully “real” emotional response occurs. At this point in the response curve, there is a predicted perceptual decoupling between the highly-realistic representation and the recognition that a robot is human or an object is real. This is an ongoing challenge in the world of human-robot interaction and VE design. However, this technical challenge might also be used to facilitate the effects of virtuality mentioned previously.

There are a few caveats to the arguments and ideas presented herein. In animals, the uncanny valley has been observed only in primates (Penn & Povinelli, 2007; Steckenfinger & Ghazanfar, 2009). However, the strategic use of VEs to provide stimuli could reveal a similar neural response in other animals. In addition, the effects of virtuality are expected to exhibit a variable effect size depending on the species chosen. Species that have high levels of what is traditionally considered animal intelligence (Matzel & Kolata, 2010) should exhibit these effects most strongly. Effects such as illusion can also be very strong in organisms with highly-specialized sensory systems, particularly given that the VE manipulation is highly specific. I propose that the key component that relates the hyper-realism of VE to the uncanny valley effect is not a set of higher-cognitive mechanisms, but rather the information held in perceptual ambiguities. It is these ambiguities and the uncanny valley effect in general that can actually be leveraged to produce illusory or space/time dilation effects.

Sensorimotor integration

An experimental apparatus that is both capable of tightly reproducing the original environment (maintaining integration) and selectively distorting it (disrupting integration) is highly useful for understanding the effects of movement disorders. Being able to conduct experiments with this level of environmental control in non-human animals allows for single cell-level contributions to behavioral variation.

Ahrens et al. (2012) have developed an innovative virtual environment for zebrafish that is customized for fish cognition and swimming behavior. Visual scenes are projected onto a screen located underneath the fish’s location (Petri dish), and consist of square gratings that move along the fish’s body from snout to tail. Importantly, the speed of visual cue presentation can be adaptively adjusted relative to swim speed. Immersion in such a context is sufficient for initiating short-term forms of motor learning (Gray, 2012). The neural populations responsible for motor learning are distributed across the brain, including the inferior olive and cerebellum. This is the expected location for motor learning consolidation, which is conserved from fishes to humans.

Zebrafish VE also allows for flexibility in the experimental setup which in turn provides a means to dissect components of the sensorimotor loop in a systematic manner. Engert (2013) has proposed two alternate interaction modes (e.g. experimental preparations) for creating illusory stimuli related to zebrafish swimming behavior. In this case, possible illusory stimuli include (but may not be limited to) oscillating visual gratings and animations that are inconsistent with an organism's perception of self-motion (Lappe et al., 1999). These type of illusions presented in an experimental setup allows for direct measurement of movement and the recording of neural responses to active behavior. The other involves immobilizing the fish and recording the neural activity associated with intended (or fictive) locomotion. In both cases, the contributions of visual stimuli, motion, and the corresponding neural response can be decoupled through an inconsistency between an organism’s self-motion and the surrounding environment.

While this effect might be explained as an experimental artifact, robot-fish interaction studies might help us further appreciate the role of conspecific-like self-motion cues in regulating how perception and action are coupled and decoupled. In the work of Marras & Porfiri (2012), biological fish were attracted to the locomotion of a robotic fish. Rather than actively decoupling sensory cues, the robot-fish interaction involves replicating the hydrodynamic and other mechanical cues of conspecific swimming behavior. While the coupling or decoupling of self-motion and behavior may be context-dependent in nature, VE and robotic studies have shown (in an almost accidental fashion) how true to context stimuli must be to elicit the proper neural responses. As we will see in the case of interactivity, neural activity associated with intentional behavior can be both a useful and important indicator of dynamic cognitive responses.

In another set of experiments in insects, virtual environments are used to dilate visual stimuli with regard to motor control. Gray et al. (2002) use the walls of a flight arena to present visual cues that mimic depth and motion to an immobilized insect. This was done in a specialized arena which is shown and discussed in Gray et al. (2002) & Seelig et al. (2010). In Seelig et al. (2010), a head-fixation task is replicated by having a fly walk on an air-supported ball concurrent with the presentation of visual stimuli. Using this type of VE design, an integrated response was found in horizontal system neurons. Using systems such as these, adaptive behaviors can be initiated in a highly-controlled environment. This not only allows for a range of behavioral regimes to be explored, but multiscale (e.g. cellular and behavioral dynamics) experimental investigations as well.

Spatial navigation

Spatial navigation is perhaps the best understood of the three featured behaviors due to our extensive knowledge of neural mechanisms at both the structural (hippocampus) and single-cell (place and grid cell) levels. Indeed, virtual environments enable the development and confirmation of sophisticated theoretical models of spatial navigation. This is exactly what was done in Holscher et al. (2005) & Harvey et al. (2009). In the Harvey et al. (2009) approach, a mouse is situated atop an air supported-spherical treadmill, and its head is fixed for purposes of in vivo measurement. The virtual environment consists of a projection-based visual display. The first-person display features a fisheye-view of a linear track with a reward at the end of the track. This experimental setup resulted in semi-realistic firing patterns for place cells, which encode locations in virtual space. The authors also found three distinct sub-threshold signatures for place fields, which in turn may allow us to confirm theoretical models of neuronal coding (Ekstrom et al., 2003).

While traditional spatial navigation experiments require very few illusory or space/time dilation-related manipulations, there is the potential to do experiments in animals where spatial relationships (and perhaps even mental representations of space-time) are warped. The work of Gershow et al. (2012) demonstrates how gradients of airborne cues can be delivered to organisms in a controlled manner using a series of microcontrollers. Some invertebrate species such as moths engage in a form of spatial navigation behavior called plume tracking. Plumes of odorants or other chemicals do not diffuse through their environmental media (e.g. air or water) in a linear fashion, and the information embedded in a plume is made highly nonlinear due to turbulent conditions. By delivering these gradients as highly laminar flows, the diversity and complexity of motor responses associated with plume tracking can be made tractable.

Interactivity

Interactivity can be defined as the ability to manipulate and adaptively respond to a wider range of objects and behaviors than would found in a non-virtual context. This is a term I am presenting here for purposes of describing a series of experiments that feature animals interacting with VE systems. This could include computer-generated stimuli or avatars. Depending on the application, this can provide either the experience of enveloping interactivity or an experience of dilating the temporal or spatial scale of perception and action.

Normand et al. (2012) use an ingenious experimental design to study interactivity between rats and humans using a technique called “beaming”. In this approach, a rat interacts with a robotic human analogue (ePuck). Humans interact with a telerobotic virtual environment system that maps behavior to ePuck that size-wise is similar to the rat’s body. To provide closed-loop feedback, the rat’s movements are then tracked and mapped to a human-like avatar in the virtual environment. The beaming approach allows for human interactions to take place at the rat’s size scale and vice versa. This also enables inter-species interactions such as the neuroanthropological studies of human-animal interaction featured in Keil & Downey (2012). Using beaming in this context might more directly address the existence of ToM within and between species.

Interactivity can also be explored using brain-machine interfaces (BMIs). BMIs share many attributes with virtual environments, and allow us to better contextualize the potential interactions between brain, behavior, and environment observed during virtual world immersion. We can look to the application of BMIs in understanding the neural mechanisms underlying grasping in non-human primates as a relevant example. In O’Doherty et al. (2011), his group introduces the brain-machine-brain interface, which uses electrophysiological signals from the motor cortex (motion planning) as input to a virtual arm that grasps virtual objects. The additional (e.g. feedback to the brain) component involves stimulation of the sensorimotor cortex that serves as haptic (e.g. touch) feedback. This set of experiments has applications to brain-controlled prosthetic devices. This brain-machine-brain interface is currently being realized in application form as the Walk Again project, which aims to enable prosthesis-wearers to engage in activities such as soccer (Yong, 2011). This includes robotic limbs that require close coordination with intentional behaviors, or even devices which record behaviorally-relevant electrical signals in one animal and uses that signal to stimulate the brain of another animal (Pais-Vieira et al., 2013).

Innovative applications of VE to understand cognitive mechanisms

There are a number of ways in which innovative virtual environments can be used to enhance our understanding of cognition and neurobiology in animals. One example involves a two-dimensional navigation system (rotation and surface walking), developed by Aronov & Tank (2014) to assess the neural activity associated with spatially-explicit exploratory behavior in rodents. In terms of building hippocampal representations, the overall shape and boundaries of a given spatial environment were found to be most important. Another example comes from Aghajan et al. (2015) assesses differences in spatial selectivity amongst hippocampal cells by comparing virtual and real-world environments. Rodents were introduced to distal visual cues in both real-world and virtual environments while performing random foraging and goal-directed tasks. Electrophysiological measurements of place cell activity (Aharoni, 2013) reveal that spatial selectivity is markedly reduced in virtual environments as compared to real-world settings. In a third (and related) example by Ravassard et al. (2013), place cell populations are shown to be partially-activated in virtual environments where spatial navigation is reliant on distal visual cues rather than sensory cues related to the vestibular system and egocentric motion (Cushman et al., 2013).

Other effects of VE on cognition

Despite these examples from specific cognitive domains, it is not clear what the effects of VE actually are. As the neural response is characterized as semi-realistic by the authors, this suggests VE may not be perceived by the animals as a real world (the virtual representation falling partially into the uncanny valley featured in Figure 1). But how does the uncanny valley-like effects become manifest in sensorimotor integration, spatial cognition, and interactivity? These are not clearly emotional behaviors, but also involve making distinctions between the real and the artificial. In the case of sensorimotor integration, the uncanny valley might involve slightly unnatural movement patterns. This could involve a detectable discontinuity in the integration of vision and touch. Such an outcome could be registered as an emotional ambiguity (e.g. what is this object?), which could in turn disrupt how the animal treats its environment. A similar outcome might be seen for spatial cognition in terms of disruptions of the spatial reference frame. Like sensorimotor integration, there is a reliance of multisensory integration as a seamless process. When this consistency is violated in terms of an animal’s locational self-awareness (e.g. where am I?), an emotional response is triggered. However, in terms of interactivity, an uncanny valley-like emotional response is more straightforward. Interactivity involves interpersonal interactions with objects and agents, and so an uncanny valley-like response occurs in much the same way as predicted by the original theory.

Figure 1. A conceptual demonstration of the uncanny valley, adapted specifically for virtual environments.

Realism (x-axis) represents the fidelity and/or resolution of this representation. Emotional valence (y-axis) represents the positive or negative emotional valence associated with a given representation of familiar objects or conspecifics. A: the first phase of the response curve, showing an initial rise in emotional valence with moderate degrees of realism. B: the second phase of the response curve, showing a dip and rebound in emotional valence at very high levels of realism. Shaded region represents hypothetical individual variation exhibited in the response. Figure adapted from the uncanny valley principle as originally proposed by Mori (1970).

Alternatively, the possibility exists that virtual worlds simply expose the diversity of responses to highly similar environmental phenomena. This is not only due to cross-talk between different cognitive processes, but also involves individual variation in learning abilities and attentional capacity. In human experiments that focus on the effects of training, subjects can be switched back and forth between virtual and real-world tasks (Rose et al., 2000). Ideally, the virtual condition should provide gains in expertise that are transferrable to the real world analogue task. A similar experimental approach might be used for disentangling the effects of a virtual environment (such as sub-threshold neuronal activations) on an animal. While it is impossible to know which interpretation is correct at this point, future experiments specifically focused on perceptual realism in animals might provide us with a clearer picture.

Key features of a virtual architecture for illusion

A virtual representation for illusion follows three sets of observations. The first involves the sensory systems that are engaged by the environment. Due to the immersive and flexible aspects of VEs, behaving animals can engage the environment in a naturalistic fashion. This includes engaging an environmental stimulus in a way analogous to behaviors such as foraging, free navigation, and mating. Therefore, considering the connections between higher-level cognition (e.g. attention) and psychophysiological phenomena (e.g. microsaccades) might be useful in selectively manipulating the input (Otero-Millan et al., 2012). In immersive contexts, the selective decoupling of vision from touch/proprioception and even audition is very important.

The use of VE systems also results in neural correlates that are distinct from real world analogues in humans, in concrete forms such as comparisons between static images and animated video (Han et al., 2005), or 2-D versus 3-D images of hand movements (Perani et al., 2001). While the sensory systems are engaged during interactions with virtual environments, areas related to multisensory integration and memory consolidation are also engaged. This is particularly true for long-lived illusions that are more than the by-product of visual after effects. As a result of this neural and sensory engagement, we should expect certain behavioral dynamics that correspond with those exhibited in the natural world. This is a consequence of behaviors being engaged in context. Ideally, an animal should produce a behavioral response to the illusion that is similar or identical to the same stimulus in the natural world. More likely (and more common with less immersive stimuli) is a behavioral shift that does not mimic the real world. This can be due to a lack of realism in the virtual stimuli, but may also be due to a lack of contextual cues.

This expected result is based on the idea that once a virtual environment reaches a certain level of realness, the brain can no longer distinguish between real and virtual stimuli. In the case of highly immersive environments, there may be an augmented effect on cognitive processes such as attention and memory (Ragan et al., 2010). Yet much like in the case of the uncanny valley, there is a regime where the brain treats virtual stimuli very differently from their physical world counterparts. Therefore, we can use informed speculation to better characterize the theoretical relationship between a continuous measure of immersion and task performance. The general variable called performance indicates a potential measure of goal-oriented behavior (e.g. swimming orientation, target accuracy) relative to a real-world control.

In the cases of space/time dilation and illusion, we can make an educated guess as to what the consequences on performance should look like. For example, the predictions for space/time dilation should show a roughly linear relationship between the degree of immersion and performance. In this case, immersion can be operationalized as the degree of exposure an organism has to a VE system. Generally, the degree of immersion increases with the level of performance. On the other hand, previous experience with a specific set of perceptual cues might change this response in certain individuals. Other types of responses might also be possible. A secondary prediction is that there should be a tendency for a flattening out of the response curve at very high and very low levels of immersion, as immediate distinctions between the real and virtual worlds become impossible.

By contrast, the predictions for illusion might involve an inverted U-shaped relationship between performance and environmental realism. As the amount of environmental realism increases from very low resolution simulation of the environment, performance should increase. Yet for very high resolution simulations, where multiple sensory modalities are simulated at very-high fidelity, performance should drop off. However, any such response would likely be expertise-dependent (C. Bohil, pers. comm.), and might be very different when the stimuli are significantly different from what is normally experienced by the organism. The Uncanny Valley effect and inverted U-shaped relationship is expected to be most prominent in cases where stimuli are unexpected with respect to experience. This can in turn interfere with higher-level mechanisms involved in perception and action.

Key features of a virtual architecture for space/time dilation

Similar questions to those that define illusory experiences in animals can also be asked in the context of space/time dilation. Depending on the degree of immersion, there are a range of sensory systems that could be engaged during space/time dilation. In mammals, this might include the visual and vestibular systems working in concert to register the location and position of the organism’s body in the environment (Fetsch et al., 2012). Unlike illusion, multisensory integration must not be disrupted over long periods of time.

The neural substrates of space/time dilation involve structures related to learning and memory, spatial cognition, and time-keeping. In mammals, these include the hippocampus (Jacobs et al., 1990) and frontostriatal-cerebellar connections (Stevens et al., 2007). In cases where space/time dilation is successfully achieved, we should expect enhanced activity in these regions. Space/Time dilation should lead to unique behavioral dynamics, very different from those expected from illusion. Highly-immersive environments should produce sped-up or slowed-down responses that are consistent with the type of space/time dilation employed. The outcome of space/time dilation is a learning effect that may reconstitute neural synchrony (Axmacher et al., 2006).

A vision for the future

While there are many unknowns in terms of how animals respond to their environment, not to mention the diversity inherent in animal brains and sensory systems, we can nevertheless selectively manipulate these variables using virtual environments. In the broader scheme of animal cognition, parallels with human cognition can be drawn in to illustrate potential neural mechanisms that might be involved in producing behavioral effects observed across a range of experiments. While these effects constitute a relatively unexplored component of animal behavior, they may lead to new discoveries in animal cognition and perhaps in the genetic substrates of conserved animal behaviors (Figure 2).

Figure 2. The role of VE in the milieu of interactions (genes, behavior, and environment) that define an organism’s world.

Elicitation of these behavioral effects is dependent on the configuration of the virtual environment itself. Unlike natural environments, virtual environments are highly stereotyped and do not include much of the noise associated with biological realism (Dennett, 2013). Nevertheless, environmental realism can be high, and findings in human experiments suggest that this is not an epiphenomenon (Blascovich & Bailenson, 2011). In addition, virtual environments are highly flexible and provide an experimental test bed for exploring the potential richness of animal perceptual, cognitive, and social behavior (Bohil et al., 2011). Since there are a range of possible design configurations for animal research-oriented VE systems, many of which can be tailored to a scientific question and organism of interest, the possibilities for further application and future research are potentially endless. Furthermore, costs can be minimized through clever design features.

Tailoring the virtual world to the perceptual specializations of a given organism would help in this regard. One example is the high critical fusion frequency (CFF) of the housefly (Healy et al., 2013). Tightly-controlled environments can be constructed by using the fly’s natural visual sampling rate as a baseline. The rate of presentation can then be systematically varied. Another example is the electrosensory and mechanosensory capabilities of sharks, rays, and certain bony fishes (Coombs et al., 2002). A VE system that models fluids and fluid dynamics in the sensory environment could enable the creation of perceptual ambiguities, which could then allow for the power of sensory illusion to be leveraged. These type of examples ultimately provides the experimentalist with a highly-controllable, selectively enriched (Nithianantharajah & Hannan, 2006), and customizable environment.

The benefit of this might be also considered in terms of gene-environment interactions (Figure 2). One way in which virtual environments might be able to assist in uncovering gene-environment interactions is by using a logic similar to that which twin studies rests upon. In twin studies, the genetic similarities of identical twins are used to control for unknown genetic variation (van Dongen et al., 2012). In a similar manner, virtual environments might be employed to control for unknown environmental noise. For experimental purposes, a random sample exposed to the same highly-controlled environment is predicted to exhibit minimal environmental variation. This should allow for the effects of the genetic background to be magnified, enabling stronger associations between genes and behavior to be made.

With the rapid adoption and increasing affordability of next-generation sequencing technologies, it is now possible to target assays of a genome in combination with genome-wide association (GWAS) studies to uncover the genetic components of a trait. What is still a mystery are the interactions between genes, behavior, and environment. Gene sequencing combined with robust environmental control can elucidate some of these interactions, while also providing insights into the ultimate processing limits of functionally-distinct neural systems.

While the link between genotype and controllable environment is more speculative, the promise of VEs for the study of animal behavior and cognition is real and the returns can be immediate. I have shown how different forms of VE have been used to elucidate and perhaps even augment animal behavior. In fact, VE might be particularly useful in understanding particularly difficult-to-define problems such as neural coding (Kumar et al., 2010) and human-animal interaction (Wilson & Barker, 2003). Overall, however, VE systems provide a flexible mode of investigation for both general and specific mechanisms that govern brain and behavior. In addition, two specific types of manipulation (illusion and space/time dilation) can be used to produce novel experimental outcomes. These effects of virtuality provide an opportunity to advance the naturalistic study of animal brain and behavior.