Based on two experiments, Thorstenson et al. (2015a) claimed that a state of sadness—induced by watching a short film clip—impairs performance on a specific perceptual task: discrimination of colors along the blue–yellow axis, but not the red–green axis. This conclusion is interesting because it is specific to a single dimension of color space; poor performance on tasks generally, or low willingness to cooperate with an experimenter, would not be a surprising effect of sadness.

In their retraction notice ( Thorstenson et al., 2015b), the authors acknowledged that their data did not justify their conclusion that impairment was specific to one aspect of color space. They also described an anomaly in the histogram of the data of Experiment 2. In our sections below entitled “A confounded comparison” and “Perceptual impairment or change in bias?”, we detail other problems with the way the experiments were conducted, the choice of stimuli, and the measures chosen. It is these problems that lead us to believe that even the revised Experiment 2 (proposed by Thorstenson et al., 2015b) will not justify their original conclusion. In our “Re-analysis of data” section, we describe a number of statistical issues that go beyond the problem mentioned in the retraction notice. In our final section, “Anomalies and strange patterns in the data”, we report some other anomalies with the dataset, which further undermine confidence in the way the experiments were carried out and in the resulting findings. We offer this analysis not only to improve the state of the literature on the perceptual effects of watching sad film clips, but also in the hope that it will lead to better work in this area in the future.

When designing an experiment comparing two conditions, one strives to make the factor of interest the only difference between the conditions. Thorstenson et al. (2015a) contrasted two film clips, one of which was intended to cause the participants to feel sad. The clips ought to have been chosen to avoid any other differences (on average) in their effect on the participants. The “sadness” clip of Experiment 1 is an excerpt from the animated Disney movie The Lion King, with an unusual lighting that gives the impression of daylight filtered through dust, while the “happiness” clip is a warmly-lit, indoor recording of the comedian Bill Cosby. The “sadness” clip used in Experiment 2 is the same Lion King excerpt, converted from color to grayscale, and the “neutral” clip is a grayscale film of sticks appearing on top of one another at different orientations (also converted from color to grayscale).

Unfortunately, differences in mean color and the color variability in these clips may have differently affected subsequent perception of blue and yellow versus red and green. For example, the contrast along the blue–yellow axis might have been greater in the sadness clips. Such a difference would result in reduced sensitivity to blue–yellow ( Krauskopf et al., 1982). The use of grayscale clips does not eliminate this issue. An analysis of the Hue, Saturation, Brightness (HSB) values in the movie files posted online indicates that the mean color of the grayscale clips is bluish-reddish, with some saturations approaching 5%. These grayscale clips, therefore, may have had differences in average color as well as color contrast that were uncontrolled. To resolve the issue, the colors displayed on the laboratory screen must be measured with a colorimeter. The authors should have made these measurements and reported them in their paper, in order to provide an indication of whether simple contrast adaptation specific to each color axis would occur when viewing the clips. In the absence of a report of these measurements or any mention of the issue, it appears that Thorstenson et al. (2015a) did not take the appropriate steps to eliminate the possibility that a classic process in color perception could explain the results.

Blue, yellow, green and red are all defined relative to a white point that, in the human visual system, is quite flexible. Just as one can adjust the white balance of a camera to fit scenes with different illumination, for humans the point considered to be the center of color space changes depending on the palette of colors that confronts us ( Webster & Leonard, 2008). Unfortunately, Thorstenson et al. displayed their test stimuli in a manner unsuited to controlling the participant’s white point. Color perception experiments typically use a neutral grey or white background to provide a white-point reference for participants, alongside the test stimulus. Thorstensen et al.’s use of full-field color without a simultaneous reference stimulus makes categorization of desaturated patches problematic. In such circumstances, the participants’ white points may be more dependent on the color content of the movie clip they viewed previously, which as mentioned above appears to have been uncontrolled. In addition, the lack of a grey or white reference stimulus may cause participants to be completely unable to judge the stimulus color more often. In such circumstances, responses may be particularly prone to influence by cognitive factors or by priming ( García-Pérez & Alcalá-Quintana, 2013).

In addition to color, there may be other confounding difference between the two types of clips. The clips likely differed in interest, action, and other features. Unfortunately, it is difficult to know whether such features might have affected participants’ color perception. It certainly is possible for such differences to bias the participants’ responses when they are uncertain of the stimulus’s color. Of course, it is almost impossible to avoid featural differences between any two particular clips. Because of this, a good experimental design would utilize a large set of clips, assess the various featural differences between the stimuli, and either match the two groups of clips carefully on their features, or model them as random effects in a mixed-effects model ( Wells & Windschitl, 1993).

Thorstenson et al. (2015a) concluded that sadness “impair[s] color perception on the blue–yellow color axis” (p. 1). But signal detection theory, which was not used, would be necessary to show whether the decrease in accuracy found was indeed due to an impairment in color perception (i.e., a decline in sensitivity along the blue–yellow axis), or whether the judgments of the sadness group were instead biased away from blue and yellow. For decades, studies of perception have used signal detection theory to distinguish between a change in perceptual ability and a change in, say, cognitive bias to press the blue or yellow button rather than the red or green one ( Green & Swets, 1966). Unfortunately, Thorstenson et al.’s plan to simply repeat Experiment 2 with minor revisions would not allow for the appropriate analysis. In Experiment 2, participants were tested in only two trials for each stimulus. Much more data would be needed to discern between a decline in the participants’ ability to discriminate the colors from a decline in the participants’ bias toward pressing the blue or yellow button (instead of the red or green). For the four-alternative categorization task used by Thorstenson et al., a multivariate extension of signal-detection theory should be used (such as general-recognition theory, Ashby & Townsend, 1986).

The analyses of Thorstenson et al. (2015a), and also the improved analyses that we have suggested above, assume stochastic independence of participants’ accuracies on the red–green stimuli and the blue–yellow stimuli. Unfortunately, however, this assumption may be unjustified. Participants’ accuracy on one axis might affect their guessing strategy on another. In Experiment 1 for example, accuracy was very high on the blue–yellow axis, suggesting that many participants may have had a clear color percept of the blue or yellow stimuli, but were less certain about the red and green stimuli. If so, when an unclear patch came up and they guessed, they may have been unlikely to guess blue or yellow, in an effort to balance their responses across the available options (many participants may have correctly guessed that the stimuli were roughly equally distributed among the four categories). This would artifactually improve performance on the red and green stimuli. Modeling this phenomenon, however, would be difficult. Even if we had access to the raw responses (rather than the summary data provided by Thorstenson et al.), it would be difficult to estimate the participants’ guessing strategy. To avoid this problem in a future version of this experiment, we suggest that Thorstenson and colleagues should consider adopting the two-alternative forced choice design ( Fechner, 1860/1966) commonly used in psychophysics.

Thorstenson et al. (2015a) are not the only researchers to have used bias-prone measures of perception to support claims that some non-perceptual state can influence perception. Firestone & Scholl (2015) provide many other examples, with useful discussion.

There are issues with the dataset ( Thorstenson et al., 2015c) that were not described in the retraction ( Thorstenson et al., 2015b) of the article. Some of these issues affect Experiment 1, which Thorstenson et al. indicated that they plan to re-publish. We describe and discuss these issues in this section, as well as the next section, “Anomalies and strange patterns in the data”.

The most important empirical claim made by Thorstenson et al. (2015a) was that there was a difference in performance between their two measures, namely color perception along the blue–yellow axis and color perception along the red–green axis. However, these authors provided no statistical test of a difference in the effect of the film clip on blue–yellow compared to red–green. This problem was discussed widely on blogs and on PubPeer (2016), and was acknowledged by Thorstenson et al. (2015b) in their retraction notice. Thorstenson et al. (2015a, p. 4) noted that the possible difference between red–green and blue–yellow color perception, such that “sadness influenced chromatic judgments about colors on the blue–yellow axis, but not those on the red–green axis,” is critical to ruling out “the possibility that sadness simply led to less effort, arousal, attention, or task engagement.” Such a difference implies a statistical interaction between the “emotion condition” and “color axis” factors. However, the authors did not report a statistical test for this interaction in either of their experiments. When we (and the authors of various blogs, such as Areshenkoff, 2015) tested this interaction with the published data, we found (code at: Holcombe et al., 2016) that it was not statistically significant: Experiment 1, F(1, 125) = 3.51, p = .06; Experiment 2: F(1, 128) = 0.40, p = .52. In their retraction notice, Thorstenson et al. (2015b) reported a z test to test the same issue (for unknown reasons, they did not use a conventional statistical interaction, but instead followed a procedure described in Rosenthal & Rosnow, 1991), which also did not yield statistical significance.

An additional potential source of error is that Thorstenson et al. (2015a) did not record the color-perception performance of their participants before the film clips were shown. It was apparently considered sufficient to randomize the participants to watch one of two film clips; presumably the reasoning was that this randomization made it unlikely that the two groups differed much in baseline performance. However, even if this assumption were to be confirmed, the two groups would likely differ somewhat at baseline, even if by only a small amount, and such a difference could have an effect on the outcome given the relatively small sample sizes involved ( Saint-Mont, 2015). It would have been useful for these differences to be measured and included in the subsequent analyses, given that Thorstenson et al.’s hypothesis was that sadness would “impair” (i.e., reduce, compared to a previous state) participants’ color perception. In addition, using a change score for each participant can increase statistical power by reducing the contribution of variation among participants to the error term.

Finally, we note that Thorstenson et al.’s (2015a) experimental design assumes the complete independence of participants’ accuracy on the two sets of stimuli (red–green and blue–yellow). We discuss a possible violation of this assumption in our “Perceptual impairment or change in bias?” section above.

While we strongly support the retraction by Thorstenson et al. (2015b) of their article ( Thorstenson et al., 2015a) on the basis of the problems they noted with Experiment 2, we maintain that the basic methodology of both of their experiments is flawed. As Thorstenson and colleagues move forward, together with others who seek to assess whether mood and other factors can influence perception, they should bring their work up to modern standards of statistics and psychophysics. Doing so for experiments like those of Thorstenson et al. would involve: (1) careful control of the visual differences between the movie clips, or, better, mood induction via non-visual stimuli such as an audio recording of a story; (2) the use of many movie clips or recordings, and mixed-effects analysis to address differences that cannot be eliminated between any two clips or recordings; (3) a baseline measurement of color perception; (4) an analysis based on signal-detection theory.

Open Science Framework: Reanalysis of Thorstenson et al.’s (2015) “Sadness Impairs Color Perception”, doi 10.17605/osf.io/kwuq4 ( Brown et al., 2016).

We have archived the R code that we used to analyze the data and generate our figures at the Open Science Framework (OSF; doi: 10.17605/osf.io/kwuq4). This code works with the original dataset files uploaded to OSF ( Thorstenson et al., 2015c), together with the patch data files that Christopher Thorstenson sent us (by “patch data”, we mean data broken down to the individual color patches tested) and that we posted at https://osf.io/sbhn9/ (Thorstenson subsequently asked us to delete two of the files, which we did).