Chinese researchers are developing emotion-sensing AI

Mar 28, 2018

In 1968, Stanley Kubrick famously highlighted the promise of emotionally-aware artificial intelligence. In 2001: A Space Odyssey, the sentient robot, HAL 9000, acknowledged a fellow crew member’s anguish by stating, “Look, Dave, I can see you’re upset about this.”

At the time, this display of acute understanding of complex human emotion by a machine was, much like the film, in the realm of science fiction. However, 50 years later, over a third of the world’s population possess devices with the potential to infer a user’s emotional state.

The capabilities of objects we carry with us every day—smartphones, fitness trackers, and watches—have entered an emotion-sensing Cambrian Explosion. They are fast becoming active participants in personalizing the world around us and influencing the content we consume on a daily basis.

“There’s this scenario I’ve postulated,” Derek Lei, General Manager of Artificial Intelligence at Emotibot, a company developing emotionally-aware bots, told TechNode. “Let’s say you’re talking with your voice assistant and you sound pretty angry. In the world of connected devices, it could communicate with your car and say: ‘Derek is pissed off this morning. He broke up with his girlfriend’.”

Based on the information it receives, your car would then be able to react by adapting its internal environment. It could play music, avoid topics on the radio dealing with breakups, and soften the lighting. These changes, along with your reaction to them, would go into your profile to further orchestrate a world just for you.

Emotion detection is fundamental if companies want to provide these kinds of personalizations. Unlike other forms of data collection, it can adapt immediately.

“When people talk about personalization, it is understanding your purchase history and knowing who you are from what you tell us,” said Lei. “But with emotion AI being deployed on smart devices, you are able to personalize at that moment.”

While emotionally-aware devices themselves may not have the ability to adapt your environment to your mood, other connected devices could. By incorporating emotion-sensing technologies into the Internet of Things (IoT), Lei says it takes personalization “to another level.” Increasing the number of these devices not only improves customization and user experience, but also provides additional sensing information from the environment, improving the accuracy of the mood detection.

The vast majority of people in China may not have access to voice assistants, but they do own smartphones. According to data from Statista, smartphone penetration in the country will reach over 50% in 2018 and almost 55% by 2020. As a result, these devices have received an increasing amount of attention for their potential emotion-sensing abilities.

Li Wenzhong is a researcher at Nanjing University studying automatic emotion detection using sensors in smartphones. Along with three other researchers, he developed MoodExplorer, an Android app that uses smartphone sensor data to detect multiple emotions simultaneously and asynchronously. The app collects environmental, contact, app usage, and activity data, correlating it with feelings manually reported by a user.

“There are other approaches for mood detection,” Li told Technode. “Different from them, our approach is based on wearable devices such as smartphones or smart bracelets. Our work tries to use the sensing data from wearable sensors and the app usage information from smartphones to achieve mood detection.”

Incorporating emotion detection into smartphones not only results in the technology being accessible but also makes sure it is mobile. Users of smartphones tend to carry them wherever they go, increasing exposure time and detection accuracy.

Li explains that the use of emotion detection in smart devices benefits both businesses and consumers. Knowing an individual’s mood enables enterprises to deploy highly-targeted advertising and make better media recommendations, including “different styles of music, movies, books, and other products.”

Conversely, businesses understanding mood metrics save customers from viewing irrelevant advertising and spending extended periods choosing a song to listen to or film to watch.

And, according to Lei, in-store experiences can be personalized according to your mood:

“If you’re a retailer and the [customer] doesn’t look very happy, what do you do? Perhaps you would want your [in-store robot] to react in a very happy way. Or maybe you want it to empathize. It really depends on how the company wants to program it.”

Social media can also enhance mood detection. When constructing a training set for developing a mood-sensing model, Li says that they rely on users to record their moods manually alongside the data collected automatically by the app. However, if sensing data is linked to social media, moods can be inferred automatically by analyzing the types of words used in user posts, negating the need for manual reporting.

It also provides a way to make sensing data more accurate. “Mood detection with smartphones and with social media can be done independently, and their results can be cross-validated to form a more accurate model,” said Li.

But the increased personalization that results from emotion-sensing technologies could bring with it a rise in data that can be collected by corporations, and in turn, a degradation of personal privacy.

Earlier this year, CCTV and Tencent Research found that over 76% of Chinese people feel AI is a threat to their privacy. While the study, which included 8000 participants, made use of a relatively small sample size, it does underscore that people in the country are increasingly aware of privacy issues.

Expressing his concern, Lei highlights a possible scenario:

“If my smart device was listening to my conversation with my girlfriend last night and it decided we had a breakup, then in the morning it could react accordingly,” he said. “Similarly, a corporation could listen in on that to sell some services.”

While some are quite happy to give up personal information on social media and on the apps they use, others aren’t. “For some people, mood itself is private, and they are reluctant to be detected by an app,” said Li.

Nonetheless, the applications provided by mood-sensing technologies are hard to ignore. While currently in the nascent stage, emotion detection shows signs of great promise and increased ubiquity in the years to come.

“It is hard to predict the future. But it is sure that in the future, smart home and wearable devices will explore more human factors such as preferences, context, and mood. Detecting mood is one step towards a smarter, personalized, humanized design for smart home techniques. Such techniques can also be useful for the development of future robots that can understand human moods,” concludes Li.