The Future of Emotion Recognition in Machine Learning

Understanding the meaning of facial expressions is an essential human survival tool, but one that generates scientific controversy. Some contend that we recognize anger most easily¹, since it may represent an existential threat; others that we recognize happiness most easily and anger most slowly², since the march of civilization has altered our priorities over time.

The lexicon of facial expression is rich in ambiguities, as facial cues may be insincere, misinterpreted, or in some way at a tangent to their intent. Experts in the understanding of facial expression remain split even on whether the Mona Lisa's enigmatic smile is sincere³ or forced⁴ (apparently a matter of context).

The ability to decipher the true intent and emotional response of a person from their facial expressions, notwithstanding their attempts to mask or deceive what they feel, is an evolutionary advantage of great interest to a range of sectors, from physicians through to marketers and political analysts. Unsurprisingly, there's a lot of money in it.

Academic research into identifying deepfake video manipulations ramped up in the 12 months prior to the 2020 US election, in response to widely-voiced concerns^6,7,8 around deceptive video content with a political agenda.

In 2019, the Computer Vision Foundation partnered with UC Berkley, Google, and DARPA to produce a system claimed to identify deepfake manipulations via the analysis of expressions in the targeted subjects⁹ — the only one, among several novel approaches¹⁰, that relies specifically on facial expressions.

Because it is more difficult to obtain a high volume of images of 'rarer' expressions for a machine learning dataset, these will inevitably be under-represented at the training stage, and will therefore be more detectable in comparison to 'stock' expressions in a deepfake video, which were trained on a higher number of expression instances.

One 2021 study leverages a deep learning method to identify facial emotion in children with autism spectrum disorder (ASD) while they are interacting with a computer screen. With 99.09% model accuracy, researchers have been able to provide ASD sufferers with timely assistance. ⁵⁴

Stanford University's Autism Glass Project¹³ uses Google's face-worn computing system to aid autism-affected subjects in understanding appropriate social cues by appending emoticons to those expressions the system is able to recognize.

Another project¹⁴ has used machine learning to develop an app to screen children for autism by running the subject's facial reactions to a movie through a behavioral coding algorithm in order to identify the nature of their responses (in comparison to a person not afflicted with ASD). The project utilizes Amazon Web Services (AWS), PyTorch, and TensorFlow.

Since autism is such an apt candidate for expression recognition technologies, there are more applications of machine learning for this cause¹⁵ than we are able to list here.

Due to increased controversy over data mining and privacy concerns¹⁶ over the last five years, the diffusion and extent of always-on driver monitoring systems has hit a number of roadblocks relative to press exposure and anticipation¹⁷ around such technologies.

Nonetheless, expression recognition is included as part of a number of available in-car systems trained by machine learning. For instance, in addition to its ability to understand if a driver is not looking at the road, not wearing a mask or is making a hands-on phone call, Affectiva's in-cabin sensing suite includes an expression-recognition component that's capable of determining if the driver is falling asleep.

This test for 'liveness' or alertness can be configured to trigger a number of warnings or other safety-related actions, which is particularly relevant as a driver control feature in commercial fleet management software.

Since monitoring user data by stealth has become such a political football in recent years, facial expression recognition technologies are on their strongest ground where the subjects are willing accomplices, such as in focus groups and other forms of beta-testing for product marketing.

Outside of such controlled environments, there are possibilities for novel use of FER in a marketing context. For example, Alfi, a Miami-based AdTech company, uses ML-based facial recognition to detect people’s emotions and deliver targeted advertising in public places in real-time. Whether you are staring at the screen in Uber or a shopping mall, Alfi’s algorithm can accurately assess a person’s age, ethnicity, and mood to show personalized content. Unsurprisingly, there is a lot of legal uncertainty and privacy concerns regarding the subject, and it’s almost certain that FER-based advertising will become widely adopted in the near future.

The rise of teleconferencing-based interaction since the advent of COVID-19 solves two major obstacles to using FER in a recruitment interview context: the negative effects¹⁹ of explicitly training a camera on someone you are talking to (solved by the user's webcam); and the more recent challenge of the subject's face being occluded by a mask and thus hindering intelligent video analysis.

Though FER is being increasingly implemented in a recruitment context^20,21, it comes with a number of caveats:

• Depending on the system, it can introduce racial bias²² that can expose employers to legal repercussion (a sub-set of wider problems around ethnicity in facial recognition²³).
• It will probably work best without user consent²⁴ — usually illegal in the EU and in many other jurisdictions.
• It seems set for further regulation^25,26, potentially compromising any current effort of investment.

Besides these considerations, an FER-based recruitment system can eventually be 'gamed': in South Korea, where FER is increasingly used in the hiring process and where a quarter of the top 131 companies either use or plan to use AI in recruiting²⁷, candidates are now schooling themselves²⁸ in anti-FER techniques.

Faced with rising ticket prices and declining audiences in 2014, a Spanish theatre experimented with charging audiences 30 cents a laugh for its comedy shows²⁹, installing emotion recognition cameras in front of each seat, and fixing a ceiling of 24 euros per customer for the more successful events.

Despite the viral spread of the project, 'metered' comedy shows have not stormed the market since, though some have speculated that the principle could be applied for other types of output, such as horror.

Since machine learning FER systems have no innate understanding of human emotion, and since they rely on our ability to accurately label expressions, it is worth considering our own limitations in this respect, since these will only be magnified in a machine learning system built around them.  

According to one professor of psychology in the US⁴¹, our own ability to interpret facial expressions amounts to much more than examining the topography of an isolated face (which constitutes the kind of data typically fed into FER systems).

Research out of the University of Glasgow has observed that essential differences between the way that people from around the world manifest emotion is even reflected in the different characteristics of popular emoticons that one nation or region might prefer over another:

A further study⁴² on this topic pitted 15 white westerners against 15 East Asian subjects, with the aim of determining whether facial cues were equivalent across such distant nations and finding that idiosyncrasies of expression do not cross borders well:

A great deal of current thought and convention around FER has its roots in the 'facial action coding system'⁴³ developed by psychologist Paul Ekman in the 1970s. Ekman's theory assigned weight to 'microfacial' expressions deemed to be indicative of hidden emotions.

Though Ekman's methodology was widely adopted (and even became the direct inspiration⁴⁴ for the investigative drama Lie To Me), it has been the subject of growing scientific criticism⁴⁵ in recent decades, and its use as a terrorist-detection tool gauged at the same level as 'flipping a coin'⁴⁶.

In spite of this, Chinese authorities have deployed emotion recognition systems based on the same principles in the Xinjiang region in western China⁴⁷.

Even evaluating some of the supposedly less ambiguous expressions becomes problematic once we leave our own locality: a smile in a country beset by corruption can be interpreted as disingenuous or even a sign of low intelligence⁴⁸; facial expressions representing pain or pleasure are quite different among diverse cultures⁴⁹; the effects of ageing on the human face make expression recognition more difficult for us and the machine systems we are informing⁵⁰; and the labeling of expression data depends more on the interpretation of scientists than the direct feedback of participants⁵¹ (who are the only ones who knew what they were feeling when the image was taken).

In addition to evidence that women in any one cultural group seem to use facial expressions differently to men⁵², the well-supported supposition that women are better able to correctly interpret facial expressions⁵³ has potential implications for the data-gathering and pre-processing stages of any FER machine learning project.