Source: Aleksandr Ozerov
Being among people of a different ethnicity, you might find it difficult to tell individuals apart in a group of faces. This has been called the “other race effect” (though because race is an unscientific concept, I will use the term ethnicity instead). But do we really perceive other-ethnicity faces differently?
Recent research has largely settled this question: the effect is small—if it exists at all—despite the fact that subjects in the experiments measurably feel that it is a large effect. It turns out that face recognition systems in our brains are far more sensitive to whether a face is familiar than whether it is of the same ethnicity as us.
The Other Race Effect is small to non-existent
Though the effect has been observed in one form or another for more than 20 years, the definitive work on this question was carried out in 2021 by graduate student Xingchen Zhou and face research pioneers Mike Burton and Rob Jenkins, all of the University of York.
Zhou and colleagues hypothesized, based on decades of past research, that face familiarity would swamp ethnicity in terms of difficulties in face memory. The group’s key innovation was to compare face familiarity and face/observer ethnicity in the same experiment. They did this by testing celebrity faces.
White and Black participants viewed the faces of celebrities and non-celebrities of White and Black ethnicities. The participants were asked to remember whether they had seen a specific image of a face or to recognize the identity of the face in different photos. In both cases, subjects were several times worse at remembering unfamiliar faces of either ethnicity than they were at remembering the faces of a person of a different ethnicity from themselves. Moreover, the actual effects of ethnicity were far smaller than the subjects themselves believed them to be. As the authors conclude, “Face recognition accuracy depends much more on whether you know the person’s face than whether you share the same race.”
Do AI systems show effects of ethnicity on face recognition?
The latest study in this vein confirmed Zhou et al.’s work and added an intriguing twist.
Kay Ritchie and colleagues at the University of Lincoln, in the U.K., performed a test of side-by-side recognition (no memory) of face photos that showed either the same person or a different person, and compared situations where the participant was either the same or a different ethnicity than the one in the photo. All face photos were unfamiliar to the participants. Ritchie et al. found that the effect of ethnicity was small to non-existent, which bolsters Zhou et al.’s conclusion.
Ritchie et al. went further by asking an artificial intelligence (AI) face recognition system called a “deep convolutional neural net” to try the same tasks. The AI system’s performance was very similar to—but even better than—human performance, which was already excellent. Moreover, the AI showed no other-ethnicity effect at all.
What it all means
I want to stress that much caution is needed in the design and training of AI systems so that ethnic biases are not reproduced in computerized systems. There is a long list of real and possible ways that deep learning systems, especially the largest and most powerful ones, reinforce existing biases against non-white ethnicities, including stereotyping, the promotion of extremist ideology, and wrongful arrest. This is illustrated, for example, in recent work by Emily Bender of the University of Washington and colleagues, including Timnit Gebru, a former co-leader of a team at Google devoted to ethics in AI, who was fired from this role in 2018 ostensibly for raising such concerns. Little has been done to address these problems on a large scale since then. Nevertheless, Ritchie et al. show that, at the much lower level of telling one face image apart from another, AI systems seem to be free of bias.
I would speculate that one’s unconscious biases regarding people of different ethnicities are at the heart of why it might feel difficult to differentiate the faces of people of other ethnicities. In reality, if you are in a place where people’s faces look different from yours, the reason that you might feel lost is simply because the faces are unfamiliar.
