Facial Recognition and Political Affiliation: New Research

Investigations have led to the development of a machine learning system that researchers believe can accurately identify an individual’s political affiliation simply by analyzing their facial features. This study, originating from the same team that previously demonstrated the potential to infer sexual orientation through facial analysis, directly confronts and skillfully sidesteps the issues of “modern phrenology,” leading to the unsettling realization that our physical appearance might reveal more personal details than we commonly assume.

The research, published this week in the Nature journal Scientific Reports, was led by Michal Kosinski of Stanford University. Kosinski initially gained attention in 2017 with findings that indicated a person’s sexual orientation could be predicted based on facial data.

The study sparked debate not necessarily regarding its methodology, but rather the very concept of detecting something inherently non-physical in this manner. However, Kosinski’s work, as he clarified at the time and subsequently, was intentionally designed to challenge such assumptions and proved as surprising and concerning to him as it did to others. The intention wasn’t to create an “AI gaydar” – in fact, it was quite the opposite. As the team explained, publication was necessary to alert others to the possibility of such a system being developed by individuals with potentially problematic intentions.

Comparable cautions are relevant here, as political affiliation, at least within the U.S. context (and currently), while not as sensitive or private as sexual preference, remains a deeply personal matter. It is not uncommon to hear of political or religious “dissidents” facing arrest or even death. If authoritarian governments could establish what they consider legitimate grounds for suspicion by stating “the algorithm identified you as a potential extremist,” rather than, for example, intercepting communications, it would significantly simplify and expand the scope of such practices.

The algorithm itself doesn’t rely on highly complex technology. Kosinski’s paper details a relatively standard process of training a machine learning system using images of over a million faces, gathered from dating websites in the U.S., Canada, and the U.K., as well as from American Facebook users. Participants in the source material self-identified as either politically conservative or liberal as part of the site’s questionnaires.

The algorithm utilized open-source facial recognition software, and after initial processing to isolate the faces (preventing background elements from influencing the results), the faces were converted into 2,048 numerical scores representing various characteristics – these aren’t necessarily easily understood features like “eyebrow color” or “nose shape,” but rather concepts more readily processed by computers.

Image Credits: Michal Kosinski / Nature Scientific Reports

The system was provided with self-reported political affiliation data, and then systematically analyzed the distinctions between the facial characteristics of individuals identifying as conservative versus those identifying as liberal. It was discovered that differences do, in fact, exist.

It’s not as straightforward as “conservatives have thicker eyebrows” or “liberals frown more.” Nor does it depend on demographic factors, which would oversimplify the process. For instance, if political party affiliation correlates with age and skin tone, prediction becomes trivial. However, while the software mechanisms employed by Kosinski are conventional, he took care to ensure the study, like its predecessor, couldn’t be dismissed as unsubstantiated.

One way to address skepticism is to have the system predict the political affiliation of individuals with matching age, gender, and ethnicity. The test involved presenting two faces, each representing a different party, and asking participants to identify which was which. Random chance would yield 50% accuracy. Humans performed only marginally better, achieving approximately 55% accuracy.

The algorithm achieved up to 71% accuracy when predicting political affiliation between individuals with similar characteristics, and 73% accuracy when presented with individuals of any age, ethnicity, or gender (while ensuring one was conservative and one was liberal).

Image Credits: Michal Kosinski / Nature Scientific Reports

While achieving three out of four correct predictions may not appear groundbreaking for contemporary AI, considering that humans struggle to surpass a coin flip, the findings suggest there is something noteworthy to consider. Kosinski has also taken steps to rule out statistical anomalies or isolated results.

The notion that your political leanings might be discernible from your face is unsettling, as while political views are not the most private information, they are reasonably considered intangible. Individuals may outwardly express their political beliefs through accessories, but one generally assumes a person’s face is neutral in that regard.

If you are curious about which specific facial features are most revealing, the system is unable to provide that information. In a related analysis, Kosinski examined several facial features (facial hair, gaze direction, various emotional expressions) to determine if they were strong predictors of political affiliation, but none yielded more than a marginal improvement over chance or human assessment.

“Head orientation and emotional expression stood out: Liberals tended to face the camera more directly, were more likely to express surprise, and less likely to express disgust,” Kosinski noted in the paper’s author notes. However, these factors only accounted for a portion of the accuracy: “That indicates that the facial recognition algorithm found many other features revealing political orientation.”

The immediate dismissal of these findings as impossible – claiming phrenology was a discredited pseudoscience – is not a productive response. It is disconcerting to contemplate the possibility, but denying a potentially significant truth does not serve us well, especially given its potential for misuse.

As with the research on sexual orientation, the purpose here is not to develop a flawless detection system, but to demonstrate its feasibility and encourage consideration of the associated risks. For example, if an oppressive theocratic government sought to suppress individuals with differing sexual orientations or political beliefs, this technology could provide a seemingly objective method for doing so. Furthermore, it can be implemented with minimal effort or direct contact with the target, unlike analyzing social media activity or purchase history (which are also revealing).

Reports have already surfaced of China utilizing facial recognition software to identify members of the Uyghur religious minority. And within our own country, this type of AI is trusted by authorities – it’s not difficult to envision law enforcement using “cutting-edge technology” to, for instance, categorize faces at a protest, stating “these 10 individuals were identified by the system as being the most liberal,” or similar.

The idea that a small team of researchers, using readily available software and a moderately sized database of faces (easily assembled by any government, if they don’t already possess one), could achieve this anywhere in the world for any purpose is deeply concerning.

“Don’t shoot the messenger,” Kosinski stated. “In my work, I am warning against widely used facial recognition algorithms. Worryingly, those AI physiognomists are now being used to judge people’s intimate traits – scholars, policymakers, and citizens should take notice.”