EEG anticipation of random high and low arousal faces and sounds

Study pre-registration

The method of this study was registered before data collection and is available here: https://osf.io/uf59a

Participants

Thirty healthy participants recruited from graduate and undergraduate students of Padova University, with normal or corrected-to-normal vision participated in the experiment. Due to the high level of noise artefact, data from two participants were removed from the analysis. All participants received 10€ for their participation in the study. The study was reviewed and approved by the ethical committee of the School of Psychology of the University of Padua (Protocol No. 2278). Before the study, participants signed an informed consent in accordance with the principles expressed in the Declaration of Helsinki.

The sample size of this study was reduced with respect to what was declared in the pre-registration due to laboratory use limitation. For this reason, we were only able to collect data from 30 participants instead of the 36 declared. This reduction clearly impacted the statistical power of our planned data analyses.

Experimental design

In the present experiment, we used two sensory categories of stimuli (i.e., visual and auditory), which were extracted from two standardized international archives. Visual stimuli consisted of pictures of 28 faces extracted from the NIMSTIM archive (Tottenham et al., 2009), whereas auditory stimuli consisted of 28 sounds chosen from the International Affective Digitized Sounds (IADS) archive (Stevenson & James, 2008). For each sensory category, the stimuli were further extracted according to their arousal value. We selected 14 neutral faces and 14 fearful faces from the NIMSTIM inventory, and 14 low- and 14 high-arousal sounds were selected from the IADS repertoire and balanced by arousal with the NIMSTIM stimuli set. These materials are available here: https://doi.org/10.6084/m9.figshare.6874871.v3 (Tressoldi et al., 2018).

Experimental paradigm. All participants were presented with two different experimental tasks, which were delivered in separate blocks (see Figure 1). The first was a passive and the second was an active task preceded by a warm-up condition where the type of stimuli was anticipated in the cue signal. Both the block presentation order and the response button were counterbalanced between subjects to avoid possible response biases. The two tasks are described in the Figure 1.

Figure 1. Experimental tasks.

The figure illustrates the sequence of events and the temporal trial structure relative to the passive (top) and the active (bottom) tasks, which were delivered in blocks. Within each task, the stimuli were randomly presented and equally distributed according to either sensory category (faces or sounds) and arousal level (high or low).

Passive task. As shown in Figure 1, at the beginning of each trial, participants were presented with a warning signal, a fixation cross presented centrally on the screen for 300 ms. After that, a fixed 1000ms blank inter-stimulus interval (ISI) was delivered, followed by a 500ms target stimulus. The target stimulus could be either the picture of a face presented on the centre of the screen or a sound delivered bilaterally through two loudspeakers, with a 50% distribution. Half of the stimuli within each category were low-arousal and the other half were high-arousal, equally distributed. Participants were told that they had to guess which kind of stimulus they would be presented with. No behavioural responses were required until they actually received the stimulus target. At target onset, participants had to discriminate between visual or auditory stimuli by pressing two different buttons on the response box. The response buttons were counterbalanced across participants. After the response, the stimulus target disappeared and a blank screen was presented for a jittered duration between 1000 and 1200 ms (inter-trial interval) before the beginning of the next trial.

Active task. In the active task, event sequence and timing were the same as those in the passive task. As illustrated in Figure 1, the only difference in comparison with the passive task was that, after the prestimulus ISI, participants were presented with a slide showing a central question mark. They were then asked to make an explicit choice about the sensory category of the upcoming stimulus by pressing the response box. This allowed us to obtain an overt behavioural measure of the anticipation of random events as the percentage of correct responses compared to chance-level for each stimulus category. Immediately after participants’ response, stimuli were presented for 500 ms. As with the passive task, the response buttons were counterbalanced across participants.

A total of 200 trials sorted by stimulus category and arousal was presented in each task, for an experiment duration of about 18 minutes. In both tasks, stimuli presentation was fully randomized. Specifically, the trial-type randomization was generated online during the ISI by using a true random number generator (TrueRNG-2™). The TrueRNG hardware uses the avalanche effect in a semiconductor junction to generate true random numbers. Randomization via an external TrueRNG device does not rely on seed-based randomization algorithms, but on current fluctuations within the device, assuring a true random distribution. The RNG was interfaced with the stimuli presentation software E-Prime™ 2.0.8.90.

EEG recordings

During the entire experiment, the EEG signal was continuously recorded using a Geodesic high-density EEG system (EGI GES-300) through a pre-cabled 128-channel HydroCel Geodesic Sensor Net (HCGSN-128) and referenced to the vertex. Scalp voltages were recorded and amplified during the entire experiment. The sampling rate was 500 Hz. The impedance was kept below 60 kΩ for each sensor. To reduce the presence of EOG artefacts, subjects were instructed to limit both eye blinks and eye movements as much as possible.

Pre-registered data analysis

Further detail about analysis, including removal of artefacts, is available from the preregistration record: https://osf.io/uf59a.

Event-related potential (ERP) analysis. The ERP analysis was performed with Matlab toolbox EEGLAB (Delorme & Makeig, 2004). The EEG was segmented offline starting from 200 ms before the cue onset and ending 300 ms after the stimulus onset. The length of the analysed epoch was 1600 ms, starting from cue onset and included 300 ms of cue/fixation cross presentation, 1000 ms of ISI, and 300 ms from stimulus onset. Our original hypothesis, driven by the results of our previous study (Duma et al., 2017), was that PPA phenomena may rely on a prestimulus sustained ERP activity, presumably resulting in the amplitude modulation of the CNV component. For this reason, our confirmatory, pre-planned analysis purposely targeted the temporal window between 300 ms and 1300 ms from trial onset.

All epochs were visually inspected to remove bad channels and rare artefacts. Artefact-reduced data were then subjected to Independent Component Analysis (Stone, 2002). All independent components were visually inspected, and those related to eye blinks, eye movements, and muscle artefacts according to their morphology and scalp distribution were discarded. The remaining components were then projected back to the electrode space to obtain cleaner EEG epochs. The remaining epochs containing excessive noise or drift (±100 μV at any electrode) were rejected. After this step, removed bad channels were reconstructed. Data were then re-referenced to the average of all electrodes, and the signal was aligned to the baseline by subtracting the mean signal amplitude in the pre-stimulus interval. Subject average and grand average ERPs were generated for each electrode site and experimental condition. To provide a spatial representation of the anticipatory ERP activity, we used the Brainstorm software’s dedicated software (Tadel et al., 2011). This allowed us to extract videos of obtained time-resolved scalp map projections of the grand average ERP activity from 0 ms to 1350ms.

Brain source analysis. Baseline corrected epochs (-200 to 0 ms) were imported to Brainstorm software (Tadel et al., 2011) with the purpose of reconstructing the cortical generators of pre-stimulus ERP activity. Conductive head volume was modelled according to the Boundary Element Method (BEM) using Open MEEG toolbox (Gramfort et al., 2010; Kybic et al., 2005). The solution space was constrained to the cerebral cortex, which was modelled as a three-dimensional grid of 15002 vertices. Furthermore, the inverse transformation was applied to the Montreal Neurological Institute (MNI) canonical mesh (http://brainmap.org/training/BrettTransform.html) of the cortex to approximate real anatomy. The EEG sensor positions were co-registered with the default anatomical mesh by employing rigid rotations and translations of digitized landmarks (anterior and posterior commissure, interhemispheric scissure, nasion, and left and right tragus). The inverse modelling was based on sLORETA implemented as a routine of the Brainstorm platform. A noise covariance matrix was generated for each participant based on the average baseline time window. For each participant, the sources were projected using the standard anatomical template provided by the MNI, and their activity was transformed in Z scores relative to the baseline.

ERP statistical analysis. We compared face vs. sound pre-stimulus ERP activity (300-1300ms) at the sensor level, including all 128 electrodes. Moreover, to further address the role of arousal in anticipation of random events, we also compared the high- and low-arousal stimuli within each sensory category. Given that a 128-channel recording system and a time resolution of 2ms (500 Hz) were employed, performing a systematic comparison of all samples present a severe risk of false positives. Hence, we used a non-parametric, mass-univariate statistical approach based on paired t-test permutations (Groppe et al., 2011) already employed by our group in previous ERP studies (Duma et al., 2017; Mento et al., 2018; Mento, 2017; Mento & Vallesi, 2016; Mento & Valenza, 2016). Specifically, 1000 Monte Carlo permutations with cluster correction were applied, using Fieldtrip (Oostenveld et al., 2011) functions, implemented in Brainstorm.

Exploratory analysis. The above-mentioned statistical analyses were preregistered and hypothesis driven. We expected to find that the presence of anticipatory effects of random stimuli was reflected in an amplitude change of the CNV component elicited during the ISI. However, it should be noted that the real nature of anticipatory brain activity of statistically unpredictable stimuli has not been fully explored. Hence, it is possible that the exact spatial and temporal locus of PPA effects could be outside the time windows expected based on previous evidence. Thus, while the use of a confirmatory approach increases the reliability and reproducibility of the results, it may exclude possible interesting effects extending over temporal windows or spatial areas originally ignored in the pre-declared analyses. For this reason, we implemented additional exploratory analyses extending the epoch of interest to the entire pre-stimulus time window (i.e., from the onset of the fixation cross to the end of stimulus presentation) with the aim of looking for possible earlier effects than those expected. To test for any data-driven difference between experimental conditions, a Support Vector Machine (SVM) analysis was performed. The SVM belongs to the class of supervised machine learning models. Given a set of training data, in which the two categories are labelled, the SVM algorithm generates a model that is able to assign new examples to one category or the other. The K-folds cross-validation technique implemented in Brainstorm (Tadel et al., 2011) has been used to validate the SVM model using the ERP of each subject as data. Cross-validation is a resampling procedure for model selection. K-fold approach ‘involves randomly dividing the set of observations into k groups, or folds, of approximately equal size. The first fold is treated as a validation set, and the method is fit on the remaining k − 1 folds’ (James et al., 2013). For each task, we contrasted face and sound stimuli and high- and low-arousal stimuli separately within each stimulus category. The results of the SVM approach are expressed as the dynamic, time-resolved evolution of the discrimination accuracy between two categories. It worth mentioning that Machine Learning (ML) approach with EEG signal shows different results based one the features of the EEG signal to classify, the feature selection criterion and the classifier algorithm (Lotte et al., 2007). Moreover, to the best of our knowledge, ML approach have never been applied to classify brain activity for statistical unpredictable stimulus presentation. For these reasons, we tested different cut-off frequency (20Hz, 10Hz, 7Hz, 5Hz, 4Hz, and 3Hz) in order to understand which frequency maximized the classification accuracy of the algorithm.

Behavioural findings

The descriptive statistics relative to the percentages of the correct predictions of faces and sounds in the active block are reported in Table 1.

The inferential statistics based on a one-tailed, one-sample t-test with the null hypothesis of 50% corresponding to the chance level yielded the following results: Faces, t-test = 2.3; p = .01; effect size d = .45; 95% CI = .019 - .85; Sounds, t-test = .97; p = .16; effect size d = .19; 95% CI = -.20 - .60.

Sensor-level ERP

The mean number of artefact-free trials per subject accepted for averaging was comparable both in the passive (Faces: 90 ± 7.79; Sounds: 92 ± 9.3) as well as in the active (Faces: 90.60 ± 10.1; Sounds: 99 ± 10) task. Likewise, the number of high- and low-arousal stimuli within each stimulus category was comparable both in the passive (Low Arousal Faces: 46.6 ± 6.34; High Arousal Faces: 43.4 ± 4.52; Low Arousal Sounds: 46.5 ± 6.6; High Arousal Sounds: 45.67 ± 6.83) and active task (Low Arousal Faces: 46.21 ± 7.6; High Arousal Faces: 44.4 ± 6.8, Low Arousal Sounds: 45.35 ± 7.57; High Arousal Sounds: 43.68 ± 6.56). This confirmed that the true-random trial sorting procedure was effective in generating an equivalent number of trials per condition, ensuring a comparable ERP signal-to-noise ratio.

As expected based on previous literature (Mento et al., 2013; Walter et al., 1964), the visual inspection of ERP waveforms revealed the presence of a reliable CNV component in all tasks. This component was represented as an increase in fronto-central negativity about 500 ms from trial onset and lasting for the full duration of the ISI duration until stimulus onset. The topographical distribution of this activity for each task is shown in Figure 2 and Figure 3 for face vs. sound and high- vs. low-arousal contrasts, respectively (see also https://doi.org/10.6084/m9.figshare.6874871.v3 (Tressoldi et al., 2018) for a time-resolved dynamic spatial representation of the CNV over the whole scalp).

Figure 2. Scalp maps of the mean activity in the 300–1300 ms prestimulus time window in passive (central column) and active (right column) tasks for face (first row) and sound (second row) stimuli after collapsing the arousal level.

Figure 3. Scalp map mean activity in the 300 – 1300ms prestimulus window in the passive (left column) and active (right column) task for faces and sound (high and low arousal, third and fourth row, respectively).

Faces vs. sounds

The visual inspection of the high scalp-resolution maps obtained by averaging the CNV time window (i.e., 300–1300ms) showed prestimulus differences in the topographical distribution between faces and sounds, with a greater fronto-central negativity for face type stimuli (Figure 2). This difference was less evident in the active block.

High- vs. low-arousal stimuli

Concerning arousal level, the scalp distribution maps displayed an increase in the fronto-central negativity for high-arousal face stimuli compared to low-arousal stimuli in the passive task, as well as a differential map localization of the CNV between high- and low-arousal sound stimuli. Notably, this brain activity pattern was not consistent across tasks. In the active task, low- and high- arousal stimuli elicited a comparable fronto-central negativity, although a larger posterior positivity was evident for low-arousal face and high-arousal sound (Figure 3).

Although the visual inspection of the spatial maps revealed qualitative topographical differences, in none of the two tasks did the non-parametric, permutation analyses performed in the pre-registered anticipatory epoch (300–1300ms from warning onset) reveal statistically significant effects, whether when comparing stimulus categories or when contrasting stimuli arousal within categories.

A possible explanation of the lack of significant statistical corroboration of visual differences may be that mass univariate statistical tests are strongly sensitive to the presence of random factors, as in the case of including the whole spatial information (128 electrodes). This may ultimately result in excluding experimental effects strictly localized in specific scalp regions or temporal windows. As mentioned above, the hypotheses about the spatio-temporal properties of PPA induced by the present experimental manipulations were driven by our previous study (Duma et al., 2017), which showed a broad fronto-central and long lasting CNV modulation in anticipation of random simulated car accidents. However, in that study, we used a low spatial resolution EEG system (32 vs. 128 channels) as well as a lower signal acquisition sampling rate (i.e., 250 vs. 500 Hz). In hindsight, this methodological difference may explain the disappearance of significant results due to a strong multiple-comparison correction. In line with this possible explanation, we observed that the fronto-central electrode activity in both tasks showed significant modulatory effects when permutations were performed without applying cluster-based correction for controlling Type 1 error.

Source level results

Our pre-registered analyses included the reconstruction of the source level maps. Although the massive cluster-based correction for multiple comparisons did not yield statistically significant observed differences between the conditions, we report the cortex activation results in Figure 4.

Figure 4. Normalized source maps of the mean activity in the pre-stimulus time window (between 300–1300ms) relatively to each task and condition.

From the cortical source map reconstruction, it is possible to identify a common activation for both tasks and stimulus categories in the superior frontal gyrus bilaterally, namely the Supplementary Motor Area that in literature has been identified as one of the most reliable neural generators of CNV (Mento et al., 2013; Mento et al., 2015; Mento, 2017). In addition to this, the cortical maps suggest the possible presence of stimulus-driven differences in primary sensory areas. In spite of this, the source-level results can be considered merely descriptive, rather than offering reliable hints for any inferential reasoning. In fact, considering the absence of statistically significant results at the sensor level, any strong interpretation from source space findings is to be discouraged as a precaution. While bearing these methodological issues in mind, one should nonetheless take into account that both sensor- and source-level descriptive results could be deemed suggestive of the possible presence of subtle effects to be further explored with ad-hoc analyses. To follow up on this indication, we performed additional exploratory statistical analyses extending over earlier temporal windows in an attempt to specifically focus on the possible presence of specific, fine-grained experimental effects.

Exploratory results

Support vector machine results. The data were analyzed with SVM algorithm by applying different exploratory frequency thresholds (20Hz, 10Hz, 7Hz, 5Hz, 4Hz, and 3Hz, with a folding parameter of 5). The best algorithm performance in terms of experimental condition discrimination was obtained by applying a 4 Hz threshold, which was then set as the target SVM frequency. As shown in Figure 6, the application of SVM in the passive task succeeded in discriminating between Face and Sound in the time window 100–300ms, reaching a discrimination accuracy value of 71.43% (see Figure 5).

Figure 5. Support Vector Machine decoding accuracy in time in the passive task.

Figure 6. Support Vector Machine Decoding accuracy in time in the active task.

In the active task, the SVM was able to correctly differentiate between high- and low-arousal sounds, reaching a decoding accuracy of 68%, but in a later temporal window, between 1100 and 1300ms (see Figure 6).

To statistically confirm the SVM findings while accounting for all spatial information, we applied cluster-based permutation statistics taking into account all 128 channels, this time targeting the specific contrasts and temporal windows indicated by the SVM algorithm as exhibiting effects.

Concerning the passive task, the results showed a bilateral posterior significant cluster of electrodes (p = .01966, cluster statistic = 39, cluster size = 28) when we compared face and sound stimuli in the 100–300ms time window. Concerning the active task, we found a significant left centro-posterior cluster (p = .047, cluster statistic = 25, cluster size = 18) when high-arousal sounds are compared to low-arousal sounds. These differences are presented in Figure 7.

Figure 7. Statistically significant differences in the 100–300ms time window.

a) in the occipital cluster for face vs sound stimuli comparison in the passive task; b) in the parietal cluster for high- vs low-arousal sound comparison in the active task.

SVM guided source level

To further address the functional nature of these phenomena, we performed the reconstruction of the cortical source of both effects described above. Both the source-level topological distribution (source mean activity between 300–1300ms) and temporal evolution over the whole prestimulus epoch are depicted in Figure 8.

Figure 8. The source activations represent the mean activity in the source space between 100–300ms (neutral fixation cross time window).

The time series below represent the time evolution of the source activation, highlighting the difference between 100–300ms (blue rectangle), respectively of the lateral occipital left (on the left) and superior temporal right (on the right).

Relative to the early anticipatory effect, we observed a higher recruitment of the occipital–parietal left cortical regions, especially the lateral occipital gyrus, which was more activated before the presentation of faces regardless of the arousal they expressed. The visual inspection of the temporal dynamics showed that this region is highly activated at about 200 ms from the presentation of the fixation cross when a face was presented. No other relevant modulations are present in the late prestimulus interval spanning the CNV timing. A second remarkable finding was a larger recruitment of neural activity in the right superior temporal gyrus, again at about 200ms when a sound was presented. Both of the above findings corroborate our original, pre-declared hypothesis that PPA brain activity is category-dependent, since it engages in the same primary sensory areas that will be activated after target presentation. In other words, the anticipation of a face pre-activates visual occipital areas lefts lateralized, whereas the anticipation of a sound pre-activates auditory temporal areas more lateralized in the right hemisphere.

When performed on the second temporal window showing late prestimulus ERP amplitude modulation between high- and low-arousal auditory stimuli in the active task, the cortical reconstruction showed that high-arousal stimuli were preceded by the recruitment of a broader cortical network as compared to low-arousal stimuli. Indeed, in both cases, the fronto-central areas around the SMA were commonly activated. However, when the fixation was actually followed by the presentation of high-arousal stimuli, we observed an additional involvement of the left superior parietal area, maximally expressed between 1100 and 1300ms from its onset (see Figure 9).

Figure 9. Mean activity in 1100–1300ms time window of high- and low-arousal sound stimuli in the active block.

Below in the figure, it is represented the time course of the superior parietal left area of the Desikan-Killaney atlas.

Underlying data

Figshare: EEG anticipation of random high and low arousal faces and sounds, https://doi.org/10.6084/m9.figshare.6874871.v4 (Tressoldi et al., 2018).

This project contains that following underlying data:

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

NIMSTIM archive materials available from http://www.macbrain.org/resources.htm. Registration and acceptance of the terms and conditions of the website must be completed in order to use the stimulus set.

International Affective Digitized Sounds (IADS) archive materials available from: https://csea.phhp.ufl.edu/media/iadsmessage.html. Those wishing to use these materials must submit the IADS Researcher Information Form and sign the IADS User Agreement.