Social Engineering Blogs

An Aggregator for Blogs About Social Engineering and Related Fields

Addressing Arguments Against Facial Expressions of Emotion

Within the past few months, many people have reached out to Humintell and asked us to comment on recent research articles that argue against facial expressions of emotion. After a lot of deliberation, Director Dr. David Matsumoto addresses those issues in the video above. 

First and foremost, I’d like to express my deep and sincere respect for all the researchers on both sides of this issue. I encourage healthy debates and more importantly, data about those debates. I think those debates are very healthy for science as well as for scientists, practitioners but most importantly, for the general public.

Dr. Matsumoto has researched and read the vast majority of all the studies that have been cited as evidence for and against the various positions that exist. In the video above, he does not get into technical issues of claims or the nature of the studies or exact data. Though he is happy to get into those discussions, they would require some knowledge of methodology. More importantly, he thinks the message that he wants to impart gets lost really easily if that path is gone down.

Dr. Matsumoto agrees with all the data he has seen from all the researchers. What he doesn’t agree with are all the interpretations or claims made about that data.

I believe data and findings are generated within the limitations of the methodologies that are used to produce that data.

If you look at the papers that argue against facial expressions of emotion, they typically don’t encompass all of the evidence for facial expressions of emotion or their universality including:

  • 100s of Judgement studies
  • Production studies
  • Studies of blind individuals
  • Studies of children and infants around the world
  • Studies of kin vs non-kin
  • Studies of family vs non-family members
  • Non-human primate studies

These types of debates have been occurring for a century. Ever since Darwin started this work and published it in 1872, these ideas have been debated hotly both in the lay public and academic discourse. Within the academic discourse, the start of these debates came from early anthropologists like Margaret Mead and Ray Birdwhistell. Those debates carried on to the 50s, 60s, and 70s. The original universality studies were conducted in the 60s and 70s. And even from the 80s these same debates and arguments have been occurring. 

To tell you the truth, the nature of the arguments made are essentially the same today as they were 30 or 40 years ago when I started being involved with them directly myself. 

A lot of the thinking that’s dominated this field and much of academia is what Dr. Matsumoto calls “logical determinism”. Logical determinism is a way of thinking that things are mutually exclusive; they’re either or, it’s this way or that way. They are either or dichotomies. Dr. Matsumoto thinks this is true for a lot of academic debates as well as much every day thinking.

What are the limitations of logical determinism?

  • Leads easily to confirmation bias. This confirmation bias exists in the way academics think about their phenomenon. It also biases the way they create studies and the way they, Dr. Matsumoto included, interpret data. 
  • Leads to what might call straw person arguments. One straw person argument heard all the time is “facial expressions of emotion are the only things that faces do” or that “they’re always reflective of an emotional state”. 

The thought that facial expressions are always reliable indicators of emotion is a straw person argument because no one who studies facial expressions of emotion today seriously believes that. 

There’s actually a recent survey of all of the most contemporary emotion researchers in the field that was published in 2016. A survey went out to about 250 of those researchers around the world and 88% of them believed there was compelling evidence for universals in any aspect of emotion. The vast majority of researchers in this field believe the existence of facial expressions of emotion but they don’t believe these extreme straw person arguments and no one does.

Faces do many, many things.

One very special thing that faces do is create facial expressions of emotion. We know that our faces can create thousands of behaviors.

We also know that our facial behavior has many other different functions such as:

  • Signal cognition and cognitive processes
  • Signal specific verbal words or phrases
  • Speech articulation
  • Signals of physical exertion or physical effort
  • Idiosyncratic things

Because of these multiple functions of facial expressions, it makes perfect sense that some experiments will find (under some conditions), that facial behaviors are not necessarily a signal of an emotion. There’s no question about that.

But what is also true is that when a true and strong emotional reaction is spontaneously triggered, and the closer that reaction is to something that is really meaningful in our lives, that will produce the impulse to create a facial expression of emotion in people all around the world.

The link between a spontaneous, strong, intense, meaningful emotional reaction and a corresponding facial expression has never been refuted by any study.

There have been many other studies about other aspects of the face, especially studies where people are judging faces. But no study that has actually elicited a meaningful, intense, emotional reaction *spontaneously* has shown otherwise. In addition, there are a lot of studies that have shown that the face does many, many other things sometimes with the same muscles we use for emotion signaling. 

It is necessary to understand the entirety of the data in terms of the complexity of the face.

What is it about the question of universality or not that gets people so heated?

Perhaps the question about universality is somehow related to how we see ourselves and humankind; whether we see humans as fundamentally similar or somehow different. It is a deep, philosophical question with no clear answers.

Although he doesn’t agree with all the interpretations that are made of the data, Dr. Matsumoto believes that we can find ways to understand the totality of the data without negating one side or the other. 

To learn more about the seven basic emotions, visit this past blog

How to Read Facial Expressions of Emotion when People are Wearing Masks

How can we identify indicators of the face, both veracity and deceptive, even when people are wearing masks?

It is true that when people wear masks, the amount of visual cues that we have (the observable surface of the face) is greatly reduced.

But not all masks are created equal. Some cover the entire mouth and lip area while some have a smaller range. So there is some variability.

Regardless of the type of mask, what you can always see is the area above the nose. Some people may have their masks right above their nose, some people have it on the tip of the nose. But you can almost always see above that.

For those of you who have had our training on reading facial expressions of emotion, there’s a lot of things that go on in the upper half of the face. Clearly you can’t see the mouth region, but you can still gather things in the upper face region.

What expressions can you see?

  • The wrinkling of the disgust will give you an appearance change between the eyes at the top of the nose.
  • The lifting of the upper eyelids in fear or surprise, but especially fear, is also visible.
  • You can see the brow and eye movements of surprise
  • You can also see inner corners of the brows going up in sadness or distress.

So even though we are not able to see the entire face, there are cues we can pick up. Of course, the fact that those cues can occur need to be on our radar or they will be missed.

If the mask is a little narrower on the face even though you cannot see exactly what’s going on on the sides of the mouth, you can still see some appearance changes a bit more laterally of the lips. So you can still kind of get whether the person is smiling or whether there’s a little contempt there. Those are also some important signals we can get.

The big message is: even if someone is wearing a mask, don’t abandon the thought that we can pick up cues in the face.

Want to learn how to read facial expressions of emotion where a portion of the face is covered?

Consider our SubX Subtle Expression Training, originally funded by and developed for US government agencies. This online training will teach you how to recognize subtle expressions – facial expressions of emotion of lower intensity that can occur as fast as 1/16th of a second.

Covid-19, Remote Communication, and Body Language

We are all facing an unprecedented challenge regarding the coronavirus pandemic. Many countries around the world are on lockdown. Businesses have been forced to close and most people have been mandated to stay at home for weeks, if not months.

As a result, communication has also undoubtedly changed. Face to face meetings, interviews and negotiations are now taking place virtually or by phone. In fact, Zoom, a cloud-based video conferencing service, has seen daily users more than quadruple during the past several weeks.

How has this shift from in-person to digital communication changed the way we read people and their body language? What are some of the long-term implications of exclusive or predominant use of virtual meetings? And finally, given the loss of contextual and nonverbal cues in remote meetings, what are some tips on how to have more effective communication?

We sat down with Humintell Director and Psychologist Dr. David Matsumoto to discuss his thoughts on the questions above.

Humans Did Not Evolve To Do 2-dimensional Communication 

The recent shift to almost exclusive digital communication has undoubtedly changed the way we read people. Dr. Matsumoto suggests that this largely stems from the fact that humans did not evolve to do 2-dimensional communication, such as through a computer screen. On the contrary, we have evolved our perceptual senses to live in a 3-dimensional world and our sense of reality is grounded in that fact.

Interactions are based on being live and in person. Being live and in person with somebody and interacting with them is what we’ve evolved to do and what we’ve learned to do all of our lives.

It is true that when communicating digitally, the visual stimulation of a person’s behavior or in their facial expressions is almost the same as being face to face, especially given the level of technology that we have now. In addition, your auditory perception (hearing ability) is present, since you can clearly hear the other person’s tone of voice. But there are clearly some substantial differences between being live vs virtual, including a reduction in nonverbal communication and nonverbal behavior.

What are some of those differences?

Real-Life vs Digital Communication Differences

In a live context when we interact with others, we receive what Dr. Matsumoto calls “full nonverbal packages”. This full package not only includes the face and facial expressions, but also includes voice, gestures, body posture, as well as leg, hand and foot movements.  In-person we also have the additional ability to sense spacing between one another, hear sounds and experience various smells.

Even though we can communicate through these remote communication technologies, there is an overall reduction in the information that we can get nonverbally.

Virtually we have much less of these observable cues. You typically cannot see the whole body and therefore cannot see many of the body movements from the waist down that you can typically see live. There are more distractions: on screen, you can see yourself and focus on what you look like. People’s backgrounds are static. It’s easy to pay less attention to the person you’re speaking to.

Digitally there is also a big data reduction in the ability to detect subtle expressions of the face. 3-dimensional (3D) settings are especially important for reading faces. Expressions are appearance changes that occur because of wrinkle patterns and how light falls on those wrinkle contours. Thus, 2-dimensional (2D) perception cuts down on the number of stimuli we can get to make inferences to read facial expressions of emotion.

What other differences are there?

Context Matters

It’s clear that seeing people’s faces is different than just having audio. However, it’s also clear that even though you may be able to see people’s faces in a remote meeting, that’s different than being live.

Perhaps the biggest way digital communication has changed the way we interact with each other, has to do with context. 

In addition to the 2D and 3D differences mentioned above, Dr. Matsumoto suggests differences between virtual and in-person interactions also has to do with the priming in our minds of the setting or context we are in. In any interaction, there is contextualization that occurs in our head that primes us to be more sensitive to certain things.

From the time we are a few months old, social and cultural rules are all contextualized and transmitted through specific contexts. Everyone around the world learns rules about how to behave, how to think, how to feel, how to act from when we’re infants. Dr. Matsumoto calls this enculturation.

By the time we’re adults, our two major contexts are typically work and home. We learn to think different, act differently, and engage with people differently in those contexts. But now that work is in our home, it’s easy to be confused. We have a different mindset and all our neural networks that tie all our learning, thinking and feelings together are disjointed. There is a lack of depth of visual cues and lack of depth of cognitive processing.

Long Term Implications

In addition to the context and priming differences, being at home means a vast reduction in human contact and interaction we normally have in our daily lives, especially with a variety of people. We probably “see” our family members and friends less frequently, if at all.

The sobering truth is that the current shelters in place around the country and world although necessary, facilitate more social isolation, the effects of which are wide and pervasive. These implications are greater if social isolation is longer.

In addition, the tracking of psychological effects of the increased use of mobile technology and social media has been studied widely. Psychologist Jean Twenge wrote an excellent article for The Atlantic entitled “Have Smartphones Destroyed a Generation?”

Rates of teen depression and suicide have skyrocketed since 2011. It’s not an exaggeration to describe iGen as being on the brink of the worst mental-health crisis in decades. Much of this deterioration can be traced to their phones.

Because it is clear social isolation and withdrawal can have extreme negative impacts, Dr. Matsumoto suggests we need to ensure that we also create time to have actual in-person human interactions. This is good not only for the relationship but keeping our mental health.

Tips on How to Have More Effective Communication

Many of us are currently trying to strike a balance while working from home. What are some tips Dr. Matsumoto has for having more effective communication while communicating virtually? See below!

  • Create a context that is conducive to have work meetings

If possible, carve out a physical workspace at home and treat that space like you would a work environment. If you have a meeting, dress like you’re going to a meeting. Context matters!

  • Check your appearance

Now that we can now see ourselves on screen, it’s good to know how we appear. See what you look like before the meeting; check your camera angle. Check your background, sounds, lighting to ensure it’s appropriate.

  • Stay focused on the interaction

It’s easy to get distracted and to think that others not paying attention

  • Use nonverbal cues to confirm understanding

Use nonverbal cues – faces and voices – along with verbal content and confirm understanding. These remote meetings may take longer, and we may need to make more effort to make sure we’re all on the same page. Ask questions. Confirm understanding verbally. Not being on the same page can have dire consequences.

  • Record and re-watch

If your service allows recording of your interactions, check out how you come off. It’s a great learning experience to see yourself.

The differences between in-person and virtual communication are pervasive and we must be realistic about our expectations and understand that there will be a reduction in nonverbal communication and nonverbal behavior.

Yet, we must also understand that our current situation is temporary. With patience and understanding we will overcome these unprecedented challenges!