Can a visual experience be biased?

That’s the question I explore in this blog post. In particular, I’ll ask whether a visual experience could be biased, in the sense of exemplifying forms of racial prejudice. I’ll suggest that the answer to this question is a tentative “yes”, and that that presents some novel challenges to how we think of both bias and visual perception.

According to a very simplistic way of thinking about visual perception, it presents the world to us just as it is: it puts us directly in touch with our environment, in a manner that allows it to play a unique, possibly foundational epistemic role. Perception in general, and visual experience with it, is sometimes treated as a kind of given: a source of evidence that is immune to the sorts of rational flaws that beset our cognitive responses to evidence. 

This approach encourages us to think of visual experience as a neutral corrective to the kinds of flaws that can arise in belief, such as bias or prejudice: there is no room in the processes that generate visual experience for the kinds of influence that cause belief to be biased or prejudiced.

But there is a tension between that view and certain facts about the subpersonal processes that support visual perception in creatures like ourselves. In particular, our visual system faces an underdetermination challenge: the light signals received by the retina fail, on their own, to determine a unique external stimulus (Scholl 2005). 

To resolve the resulting ambiguity, the visual system must rely on prior information about the environment, and likely stimuli within it. But those priors are not fixed and immutable: the visual system updates them in light of previous experience (Chalk et al 2010, Chun &Turk-Browne 2008). In this way, the visual system learns from the idiosyncratic course that the individual takes through the world.

Equally, the visual system is overwhelmed with possible input: the information available from the environment at any one moment far surpasses what the brain can process (Summerfield & Egner 2009). It must selectively attend to certain objects or areas within the visual field, in order to prioritise the highest value information.

Pre-existing expectations and priorities determine the salience of information within a given scene. The nature and content of the visual experience you are having at any moment in part depends on the relative value you place on the information in your environment.

We perceive the world, then, in light of our prior expectations, and past exposure to it. Those processes of learning and adaptation, of developing skills that fit a particular environmental context, leave visual perception vulnerable to a kind of visual counterpart to bias: we do not come to the world each time with fresh eyes. If we did, we would see less accurately and efficiently than we do.

Cognitive biases often emerge as a response to particular environmental pressures: they persist because they lend some advantage in certain circumstances, but come at the expense of sensitivity to certain other information (Kahneman & Tversky 1973). 

Similarly, the capacity of the visual system to develop an expertise within a particular context can restrict its sensitivity to certain sorts of information. We can see this kind of structure in the specialist abilities we develop to see faces.

You might naturally think that we perceive high-level features of faces, such as the emotion they display or the racial category they belong to, not directly, but only in virtue of, or perhaps via some kind of sub-personal inference from, their lower-level features: the arrangement of facial features, for instance, or the colour and shading that let us pick out those features. 

In fact, there’s good evidence that we perceive the social category of a face, or the emotion it displays directly. For instance, we demonstrate “visual adaptation” to facial emotion: after seeing a series of angry faces, a neutral face appears happy. And those adaptation effects are specific to the gender and race of the face,suggesting that these categories of faces may be coded for my different neural populations (Jaquet, Rhodes, & Hayward 2007, 2008; Jacquet & Rodes 2005, Little, DeBruine, & Jones 2005).

Moreover, our skills at face perception seem to be systematically arranged along racial lines: most people are better at recognizing own-race and dominant-race faces, (Meissner & Brigham 2001), the result of a process of specialisation that emerges over the first 9 months of life as infants gradually lose the capacity to recognize faces of different or non-dominant races (Kelly et al. 2007). 

A White adult in a majority white society will generally be better at recognising other white faces than Black or Asian faces, for instance, whereas a Black person living in a majority Black society will conversely be less good at recognising White than Black faces.  This extends to the identification of emotion from faces, as well as their recognition: subjects are more accurate at identifying the emotion displayed on dominant or same-race faces than other-race faces (Elfenbeim & Ambady 2002).

One way of understanding this profile of skills is to think of faces as arranged within a multidimensional “face space” depending on their similarity to one another. We hone our perceptual capacities within that area of face space to which we have most exposure. That area of face space becomes, in effect, stretched, allowing for finer grained distinctions between faces. (Valentine 1991; Valentine, Lewis and Hills 2016). 

The greater “distance” between faces in the area of face space in which we are most specialised renders those faces more memorable and easier to distinguish from one another. Another way of thinking of this is in terms of “norm-based coding” (Rhodes and Leopold 2011): faces are encoded relative to the average face encountered. Faces further from the norm suffer in terms of our visual sensitivity to the information they carry.

On the one hand, it isn’t hard to see how this kind of facial expertise could help us extract maximal information from the faces we most frequently encounter.  But the impact of this “same-race face effect” more generally is potentially highly problematic: a White person in a majority White society will be less likely to accurately recognise a Black individual, and less able to accurately perceive their emotions from their face. 

That diminution of sensitivity to faces of different races paves the way for a range of downstream impacts. Since the visual system fails to advertise this differential sensitivity, the individual is liable to reason as though they have read their emotions with equal perspicuity, and to draw conclusions on that basis (that the individual feels less perhaps, when the emotion in question is simply visually obscure to them). 

Relatedly, the lack of information extracted perceptually from the face makes it more likely that the individual will fill that shortfall of information by drawing on stereotypes about the relevant group: that Black people are aggressive, for instance, (Shapiro et al. 2009; Brooks and Freeman 2017). And restrictions on the ability to accurately recall certain faces will bring with them social costs for those individuals.

Compare this visual bias to someone writing a report about two individuals, one White and one Black. The report about the White person is detailed and accurate, whilst the report on the Black person is much sparser, lacking information relevant to downstream tasks. In such a case, we would reasonably regard the report writer as biased, particularly if their report writing reflected this kind of discrepancy between White and Black targets more generally. If the visual system displays a structurally similar bias in the information it provides us with, should we regard it, too, as biased?

To answer that question, we need to have an account of what it is for anything to be biased, be it a visual experience, a belief, or a disposition to behave or reason in some way or other. We use ‘bias’ in many different ways. In particular, we need to distinguish here what I call formal bias from prejudicial bias. In certain contexts, a bias may be relatively neutral. 

A ship might be deliberately given a bias to list towards the port side, for instance, by uneven distribution of ballast. Similarly, any system that resolves ambiguity in incoming signal on the basis of information it has encountered in the past is biased by that prior information. But that’s a bias that, for the most part, enhances rather than detracts from the accuracy of the resulting judgements or representations.  We could call biases of this kind formal biases. 

Bias also has another, more colloquial usage, according to which it picks out something distinctively negative, because it indicates an unfair or disproportionate judgement, a judgement subject to an influence that is distinctively illegitimate in some way. Bias in this sense often involves undue influence by demographic categories, for instance. 

We might describe an admissions process as biased in this way if it disproportionately excludes working-class candidates, or women, or people with red hair. We can call bias of this kind prejudicial bias. 

The visual system is clearly capable of exhibiting the first kind of bias. As a system that systematically learns from past experiences in order to effectively prioritise and process new information, it is a formally biased system. 

Similarly, the same-race face effect in face perception involves the systematic neglect of certain information as the result of task-specific expertise. That renders it an instance of formal bias.

To decide whether this also constitutes an instance of prejudicial bias, we need to ask: is that neglect of information illegitimate? And if so, on what grounds? Two difficulties present themselves at this juncture. 

The first is that we are, for the most part, not used to assessing the processes involved in visual perception as legitimate, or illegitimate (though that has come under increasing pressure recently, in particular in Siegel (2017).) We need to develop a new set of tools for this kind of critique. 

The second difficulty is the way in which formal bias, including the development of perceptual expertise of the kind demonstrated in the same race face effect, is a virtue of visual perception. It makes visual perception not just efficient, but possible. Acknowledging that can seem to restrict our ability to condemn the bias in question as not just formal, but prejudicial.

This throws us up against the question: what is the relationship between formal and prejudicial bias? Formal bias is often a virtue: it allows for the more efficient extraction of information, by drawing on relevant post information. 

Prejudicial bias on the other hand is a vice: it limits the subjects’ sensitivity to relevant information in a way that seems intuitively problematic. What are the circumstances under which the virtue of formal bias becomes the vice of prejudicial bias?

In part, this seems to depend on the context in which the process in question is deployed, and the task at hand. The virtues of formal biases rely on stability in both the individual’s environment and goals: that’s when reliance on past information and expertise developed via consistent exposure to certain stimuli is helpful. 

The same-race face effect develops as the visual system learns to extract information from those faces it most frequently encounters. The resulting expertise cannot adapt at the same pace as our changing, complex social goals across a range of contexts. 

As a result, this kind of formal perceptual expertise results in a loss of important information in certain contexts: an instance of prejudicial bias. If that’s right, then the distinction between formal and prejudicial bias isn’t one that can be identified just by looking at a particular cognitive process in isolation, but only by looking at that process across a dynamic set of contexts and tasks.

References

Brooks, J. A., & Freeman, J. B. (2017). Neuroimaging of person perception: A social-visual interface. Neuroscience Letters. https://doi.org/10.1016/j.neulet.2017.12.046

Chalk, M. S., & Series, A. R. (2010). Rapidly learned stimulus expectations alter perception of motion. Journal of Vision, 108(2), 1–18.

Chun, M. M., & Turk-Browne, N. B. (2008). Associative Learning Mechanisms in Vision. Oxford University Press. Retrieved from http://www.oxfordscholarship.com/view/10.1093/acprof:oso/9780195305487.001.0001/acprof-9780195305487-chapter‑7

Elfenbein, H. A., & Ambady, N. (2003). When familiarity breeds accuracy: cultural exposure and facial emotion recognition. Journal of Personality and Social Psychology, 85(2), 276.

Jaquet, E., & Rhodes, G. (2008). Face aftereffects indicate dissociable, but not distinct, coding of male and female faces. Journal of Experimental Psychology. Human Perception and Performance, 34(1), 101–112. https://doi.org/10.1037/0096–1523.34.1.101

Jaquet, E., Rhodes, G., & Hayward, W. G. (2007). Opposite aftereffects for Chinese and Caucasian faces are selective for social category information and not just physical face differences. The Quarterly Journal of Experimental Psychology, 60(11), 1457–1467. https://doi.org/10.1080/17470210701467870

Jaquet, E., Rhodes, G., & Hayward, W. G. (2008). Race-contingent aftereffects suggest distinct perceptual norms for different race faces. Visual Cognition, 16(6), 734–753. https://doi.org/10.1080/13506280701350647

Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80, 237–251.

Kelly, D. J., Quinn, P. C., Slater, A. M., Lee, K., Ge, L., & Pascalis, O. (2007). The other-race effect develops during infancy: Evidence of perceptual narrowing. Psychological Science, 18(12), 1084–1089.

Little, A. C., DeBruine, L. M., & Jones, B. C. (2005). Sex-contingent face after-effects suggest distinct neural populations code male and female faces. Proceedings of the Royal Society B: Biological Sciences, 272(1578), 2283–2287. https://doi.org/10.1098/rspb.2005.3220

Meissner, C. A., & Brigham, J. C. (2001). Thirty years of investigating the own-race bias in memory for faces: A meta-analytic review. Pyschology, Public Policy and Law, 7(1), 3–35.

Rhodes, G., & Leopold, D. A. (2011). Adaptive Norm-Based Coding of Face Identity. https://doi.org/10.1093/oxfordhb/9780199559053.013.0014

Scholl, B. J. (2005). Innateness and (Bayesian) visual perception. In P. Carruthers (Ed.), The Innate Mind: Structure and Contents(p. 34). New York: Oxford University Press.

Shapiro, J. R., Ackerman, J. M., Neuberg, S. L., Maner, J. K., Becker, D. V., & Kenrick, D. T. (2009). Following in the Wake of Anger: When Not Discriminating Is Discriminating. Personality & Social Psychology Bulletin, 35(10), 1356–1367. https://doi.org/10.1177/0146167209339627

Siegel, S. (2017). The Rationality of Perception. Oxford University Press

Summerfield, C., & Egner, T. (2009). Expectation (and attention) in visual cognition. Trends in Cognitive Science, 13(9), 403–409.

Valentine, T. (1991). A unified account of the effects of distinctiveness, inversion, and race in face recognition. The Quarterly Journal of Experimental Psychology, 43(2), 161–204.

Valentine, T., Lewis, M. B., & Hills, P. J. (2016). Face-space: A unifying concept in face recognition research. The Quarterly Journal of Experimental Psychology, 69(10), 1996–2019.