Open Access

The Effects of Robot Voices and Appearances on Users’ Emotion Recognition and Subjective Perception

Grado Department of Industrial and Systems Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA

Search for more papers by this author

Jaclyn Barnes

Department of Computer Science, Michigan Technological University, Houghton, MI, USA

Search for more papers by this author

Jiayuan Dong

Grado Department of Industrial and Systems Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA

Search for more papers by this author

Chung Hyuk Park

Department of Biomedical Engineering, Department of Computer Science, The George Washington University, Washington DC, USA

Search for more papers by this author

Ayanna Howard

Department of Electrical and Computer Engineering, The Ohio State University, Columbus, OH, USA

Search for more papers by this author

, and

Myounghoon Jeon

Grado Department of Industrial and Systems Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA

Department of Computer Science, Michigan Technological University, Houghton, MI, USA

E-mail Address: myounghoonjeon@vt.edu

Corresponding author.

Search for more papers by this author

https://doi.org/10.1142/S0219843623500019Cited by:8 (Source: Crossref)

Abstract

As the influence of social robots in people’s daily lives grows, research on understanding people’s perception of robots including sociability, trust, acceptance, and preference becomes more pervasive. Research has considered visual, vocal, or tactile cues to express robots’ emotions, whereas little research has provided a holistic view in examining the interactions among different factors influencing emotion perception. We investigated multiple facets of user perception on robots during a conversational task by varying the robots’ voice types, appearances, and emotions. In our experiment, 20 participants interacted with two robots having four different voice types. While participants were reading fairy tales to the robot, the robot gave vocal feedback with seven emotions and the participants evaluated the robot’s profiles through post surveys. The results indicate that (1) the accuracy of emotion perception differed depending on presented emotions, (2) a regular human voice showed higher user preferences and naturalness, (3) but a characterized voice was more appropriate for expressing emotions with significantly higher accuracy in emotion perception, and (4) participants showed significantly higher emotion recognition accuracy with the animal robot than the humanoid robot. A follow-up study ( $N = 10$ ) with voice-only conditions confirmed that the importance of embodiment. The results from this study could provide the guidelines needed to design social robots that consider emotional aspects in conversations between robots and users.

Keywords:

References

1. J. A. Bachorowski and M. J. Owren, Sounds of emotion: Production and perception of affect-related vocal acoustics, Ann. NY. Acad. Sci. 1000(1) (2003) 244–265. Crossref, Google Scholar
2. T. Bänziger, D. Grandjean and K. R. Scherer, Emotion recognition from expressions in face, voice, and body: The multimodal emotion recognition test (MERT), Emotion 9(5) (2009) 691. Crossref, Web of Science, Google Scholar
3. J. Barnes, S. M. F. Hosseini, M. Jeon, C. H. Park and A. M. Howard, The influence of robot design on acceptance of social robots, in Paper presented at the 2017 14th Int. Conf. Ubiquitous Robots and Ambient Intelligence (URAI) (Jeju, Korea (South), 2017), pp. 51–55. Crossref, Google Scholar
4. J. A. Barnes, C. H. Park, A. Howard and M. Jeon, Child-robot interaction in a musical dance game: An exploratory comparison study between typically developing children and children with autism, Int. J. Hum. Comput. Interact. 37(3) (2021) 249–266. Crossref, Web of Science, Google Scholar
5. J. Barnes, E. Richie, Q. Lin, M. Jeon and C. H. Park, Emotive voice acceptance in human–robot interaction, in Paper presented at the Proc. 24th Int. Conf.Auditory Display (Houghton, MI, 2018), pp. 271–274. Google Scholar
6. J. S. Beer and K. N. Ochsner, Social cognition: a multi level analysis, Brain Res. 1079(1) (2006) 98–105. Crossref, Google Scholar
7. R. Bevill, C. H. Park, H. J. Kim, J. W. Lee, A. Rennie, M. Jeon and A. M. Howard, Interactive robotic framework for multi-sensory therapy for children with autism spectrum disorder, in 2016 11th ACM/IEEE Int. Conf. Human-Robot Interaction (HRI) (IEEE, New York, 2016), pp. 421–422. Crossref, Google Scholar
8. P. Birkholz, L. Martin, K. Willmes, B. J. Kröger and C. Neuschaefer-Rube, The contribution of phonation type to the perception of vocal emotions in German: An articulatory synthesis study, J. Acoust. Soc. Am. 137(3) (2015) 1503–1512. Crossref, Web of Science, Google Scholar
9. D. Bryant, J. Bornstein and A. Howard, Why should we gender? The effect of robot gendering and occupational stereotypes on human trust and perceived competency, in Paper presented at the ACM/IEEE Int. Conf. Human-Robot Interaction (HRI) (Cambridge, UK, 2020), pp. 13–21. Google Scholar
10. M. Cakmak, G. Hoffman and A. Thomaz, Computational human-robot interaction, Found. Trends Robot. 4(2–3) (2016) 104–223. https://doi.org/10.1561/2300000049. Crossref, Google Scholar
11. R. A. Calvo and S. D’Mello, Affect detection: An interdisciplinary review of models, methods, and their applications, IEEE Trans. Affect. Comput. 1(1) (2010) 18–37. https://doi.org/10.1109/t-affc.2010.1. Crossref, Web of Science, Google Scholar
12. M. J. E. Coeckelbergh and I. Technology, Moral appearances: Emotions, robots, and human morality, 12(3) (2010) 235–241. Google Scholar
13. D. Conti, C. Cirasa, S. Di Nuovo and A. Di Nuovo, “Robot, tell me a tale!”: A social robot as tool for teachers in kindergarten, Interact. Stud. 21(2) (2020) 220–242. Crossref, Web of Science, Google Scholar
14. D. T. Cordaro, D. Keltner, S. Tshering, D. Wangchuk and L. M. Flynn, The voice conveys emotion in ten globalized cultures and one remote village in Bhutan, Emotion 16(1) (2016) 117. Crossref, Web of Science, Google Scholar
15. C. Darwin and P. Prodger, The Expression of the Emotions in Man and Animals (Oxford University Press, USA, 1998). Crossref, Google Scholar
16. J. Decety, The neurodevelopment of empathy in humans, Develop. Neurosci. 32(4) (2010) 257–267. Crossref, Web of Science, Google Scholar
17. M. Díaz, N. Nuño, J. Saez-Pons, D. E. Pardo and C. Angulo, Building up child-robot relationship for therapeutic purposes: From initial attraction towards long-term social engagement, in Face and Gesture 2011 (IEEE, New York, 2011), pp. 927–932. Crossref, Google Scholar
18. C. F. DiSalvo, F. Gemperle, J. Forlizzi and S. B. Kiesler, All robots are not created equal: The design and perception of humanoid robot heads, in Proc. Conf. Designing Interactive Systems: Processes, Practices, Methods, and Techniques, DIS (London, UK, 2002), pp. 321–326. Crossref, Google Scholar
19. P. Ekman, An argument for basic emotions, Cognit. Emot. 6(3–4) (1992) 169–200. Crossref, Web of Science, Google Scholar
20. P. Ekman and D. Cordaro, What is meant by calling emotions basic, Emot. Rev. 3(4) (2011) 364–370. Crossref, Google Scholar
21. N. Epley, A. Waytz and J. T. Cacioppo, On seeing human: A three-factor theory of anthropomorphism, Psychol. Rev. 114(4) (2007) 864–886. Crossref, Web of Science, Google Scholar
22. F. Eyssel, L. Ruiter, D. Kuchenbrandt, S. Bobinger and F. Hegel, ‘If you sound like me, you must be more human’: On the interplay of robot and user features on human-robot acceptance and anthropomorphism, in Paper presented at the 2012 7th ACM/IEEE Int. Conf. Human-Robot Interaction (HRI) (Boston, MA, 2012), pp. 125–126. Google Scholar
23. F. Eyssel, D. Kuchenbrandt, F. Hegel and L. de Ruiter, Activating elicited agent knowledge: How robot and user features shape the perception of social robots, in Paper presented at the 2012 IEEE RO-MAN: The 21st IEEE Int. Symp. Robot and Human Interactive Communication (2012). Crossref, Google Scholar
24. S. M. FakhrHosseini, S. Hilliger, J. Barnes, M. Jeon, C. H. Park and A. M. Howard, Love at first sight: Mere exposure to robot appearance leaves impressions similar to interactions with physical robots, in Paper presented at the 2017 26th IEEE Int. Symp. Robot and Human Interactive Communication (RO-MAN) (2017). Google Scholar
25. S. M. FakhrHosseini, D. Lettinga, E. Vasey, Z. Zheng, M. Jeon, C. H. Park and A. M. Howard, Both “look and feel” matter: Essential factors for robotic companionship, in Paper presented at the 2017 26th IEEE Int. Symp. Robot and Human Interactive Communication (RO-MAN) (2017). Google Scholar
26. J. Feldmaier, M. Stimpfl and K. Diepold, Development of an emotion-competent SLAM agent, in Paper presented at the Proc. Companion of the 2017 ACM/IEEE Int. Conf. Human-Robot Interaction (2017). Crossref, Google Scholar
27. K. Fischer, M. Jung and L. C. Jensen, Emotion expression in HRI: When and why, in Paper presented at the 2019 14th ACM/IEEE Int. Conf. Human-Robot Interaction (HRI) (Daegu, Korea (South), 2019), pp. 29–38. Crossref, Google Scholar
28. T. Fong, I. Nourbakhsh and K. Dautenhahn, A survey of socially interactive robots, Robot. Auton. Syst. 42(3–4) (2003) 143–166. Crossref, Web of Science, Google Scholar
29. M. R. Fraune, S. Sherrin, S. Sabanović and E. R. Smith, Rabble of robots effects: Number and type of robots modulates attitudes, emotions, and stereotypes, in Proc. Tenth Annual ACM/IEEE Int. Conf. Human-Robot Interaction (Portland, OR, 2015), pp. 109–116. Crossref, Google Scholar
30. C. D. Frith and U. Frith, The neural basis of mentalizing, Neuron 50(4) (2006) 531–534. Crossref, Web of Science, Google Scholar
31. J. Ham, R. H. Cuijpers and J. J. Cabibihan, Combining robotic persuasive strategies: The persuasive power of a storytelling robot that uses gazing and gestures, Int. J. Soc. Robot. 7(4) (2015) 479–487. Crossref, Web of Science, Google Scholar
32. K. S. Haring, K. Watanabe and C. Mougenot, The influence of robot appearance on assessment, in 2013 8th ACM/IEEE Int. Conf. Human-Robot Interaction (HRI) (IEEE, New York, 2013), pp. 131–132. Crossref, Google Scholar
33. M. Jeon and I. A. Rayan, The effect of physical embodiment of an animal robot on affective prosody recognition, in Paper presented at the Int. Conf. Human-Computer Interaction (HCII) (Orlando, FL, 2011), pp. 523–532. Crossref, Google Scholar
34. M. F. Jung (2017). Affective grounding in human-robot interaction, in Paper presented at the 2017 12th ACM/IEEE Int. Conf. Human-Robot Interaction (HRI) (2017). Crossref, Google Scholar
35. T. Kishi, T. Kojima, N. Endo, M. Destephe, T. Otani, L. Jamoneand and S. Cosentino, Impression survey of the emotion expression humanoid robot with mental model based dynamic emotions, in Paper presented at the 2013 IEEE Int. Conf. Robotics and Automation (ICRA) (IEEE, New York, 2013), pp. 1663–1668. Crossref, Google Scholar
36. S. Ko, X. Liu, J. Mamros, E. Lawson, H. Swaim, C. Yao and M. Jeon, The effects of robot appearances, voice types, and emotions on emotion perception accuracy and subjective perception on robots, in Int. Conf. Human-Computer Interaction (Springer, Cham, 2020), pp. 174–193. Crossref, Google Scholar
37. M. Kraus, J. Kraus, M. Baumann and W. Minker, Effects of gender stereotypes on trust and likability in spoken human-robot interaction, in Paper presented at the Proc. Eleventh Int. Conf. Language Resources and Evaluation (LREC-2018) (Miyazaki, Japan, 2018), pp. 112–118. Google Scholar
38. S. S. Kwak, Y. Kim, E. Kim, C. Shin and K. Cho, What makes people empathize with an emotional robot?: The impact of agency and physical embodiment on human empathy for a robot, in 2013 IEEE RO-MAN (IEEE, New York, 2013, August), pp. 180–185. Crossref, Google Scholar
39. J. D. Lewis and A. Weigert, Trust as a social reality, Soc. Forc. 63(4) (1985) 967–985. Crossref, Web of Science, Google Scholar
40. J. Li, The benefit of being physically present: A survey of experimental works comparing copresent robots, telepresent robots and virtual agents, Int. J. Hum.-Comput. Stud. 77 (2015) 23–37. Crossref, Web of Science, Google Scholar
41. Z. T. Liu, F. F. Pan, M. Wu, W. H. Cao, L. F. Chen, J. P. Xu and M. T. Zhou, A multimodal emotional communication based humans-robots interaction system, in 2016 35th Chinese Control Conf. (CCC) (IEEE, New York, 2016), pp. 6363–6368. Crossref, Google Scholar
42. R. Lowe, E. Barakova, E. Billing and J. Broekens, Grounding Emotions in Robots–An Introduction to the Special Issue (Sage Publications, London, England, 2016). Crossref, Google Scholar
43. C. McGinn and I. Torre, Can you tell the robot by the voice? An exploratory study on the role of voice in the perception of robots, in Paper presented at the 2019 14th ACM/IEEE Int. Conf. Human-Robot Interaction (HRI) (Daegu, Korea (South), 2019), pp. 211–221. Crossref, Google Scholar
44. R. L. Mitchell and L. H. Phillips, The overlapping relationship between emotion perception and theory of mind, Neuropsychologia 70 (2015) 1–10. Crossref, Web of Science, Google Scholar
45. S. Nabe, S. J. Cowley, T. Kanda, K. Hiraki, H. Ishiguro and N. Hagita, Robots as social mediators: Coding for engineers, in Paper presented at the ROMAN 2006-The 15th IEEE Int. Symp. Robot and Human Interactive Communication (Hatfield, UK, 2006), pp. 384–390. Crossref, Google Scholar
46. C. Nas, U. Foeh, S. Brave and M. Somoza, The effects of emotion of voice in synthesized and recorded speech, in Paper presented at the Proc. AAAI Symp. Emotional and Intelligent II: The Tangled Knot of Social Cognition (2001). Google Scholar
47. S. Nonaka, K. Inoue, T. Arai and Y. Mae, Evaluation of human sense of security for coexisting robots using virtual reality. 1st report: Evaluation of pick and place motion of humanoid robots, in Paper presented at the IEEE Int. Conf. Robotics and Automation, Proc. ICRA’04, Vol. 3. (New Orleans, LA, 2004), pp. 2770–2775. Crossref, Google Scholar
48. A. Ortony, Are all “basic emotions” emotions? a problem for the (basic) emotions construct, Perspect. Psychol. Sci. 17(1) (2021) 41–61. https://doi.org/10.1177/1745691620985415. Crossref, Web of Science, Google Scholar
49. A. Pereira, C. Martinho, I. Leite and A. Paiva, iCat, the chess player: The influence of embodiment in the enjoyment of a game, in Proc. 7th Int. Joint Conf. Autonomous Agents and Multiagent Systems, Vol. 3 (Estoril, Portugal, 2008), pp. 1253–1256. Google Scholar
50. M. L. Phillips, Understanding the neurobiology of emotion perception: Implications for psychiatry, Br. J. Psychiatr. 182(3) (2003) 190–192. Crossref, Google Scholar
51. R. Plutchik, A general psychoevolutionary theory of emotion, in Theories of Emotion (Academic Press, Cambridge, 1980), pp. 3–33. Crossref, Google Scholar
52. R. Reisenzein, E. Hudlicka, M. Dastani, J. Gratch, K. Hindriks, E. Lorini and J.-J. C. Meyer, Computational modeling of emotion: Toward improving the inter- and intradisciplinary exchange, IEEE Trans. Affective Comput. 4(3) (2013) 246–266. https://doi.org/10.1109/t-affc.2013.14. Crossref, Web of Science, Google Scholar
53. E. Rodriguez-Lizundia, S. Marcos, E. Zalama, J. Gómez-García-Bermejo and A. Gordaliza, A bellboy robot: Study of the effects of robot behaviour on user engagement and comfort, Int. J. Human-Comput. Stud. 82 (2015) 83–95. Crossref, Web of Science, Google Scholar
54. J. Russel, A circumplex model of emotions, J. Pers. Soc. Psychol. 39 (1980) 1161–1178. Crossref, Google Scholar
55. J. A. Russell (2017). Cross-cultural similarities and differences in affective processing and expression, in Emotions and Affect in Human Factors and Human-Computer Interaction, eds. M. Jeon (Academic Press, Cambridge), pp. 123–141. Crossref, Google Scholar
56. S. Saint-Aimé, B. Le-Pevedic, D. Duhaut and T. Shibata, EmotiRob: Companion robot project, in Paper presented at the RO-MAN 2007: The 16th IEEE Int. Sym. Robot and Human Interactive Communication (Jeju, Korean (South), 2007), pp. 919–924. Crossref, Google Scholar
57. K. Sakai, Y. Nakamura, Y. Yoshikawa and H. Ishiguro, Effect of robot embodiment on satisfaction with recommendations in shopping malls, IEEE Robot. Autom. Lett. 7(1) (2021) 366–372. Crossref, Web of Science, Google Scholar
58. T. S. S. Schilhab, Anthropomorphism and mental state attribution, in Animal Behavior (Academic Press, Cambridge, 2002). Google Scholar
59. A. Schirmer and R. Adolphs, Emotion perception from face, voice, and touch: Comparisons and convergence, Trends Cogn. Sci. 21(3) (2017) 216–228. https://doi.org/10.1016/j.tics.2017.01.001. Crossref, Web of Science, Google Scholar
60. S. H. Seo, D. Geiskkovitch, M. Nakane, C. King and J. E. Young, Poor thing! Would you feel sorry for a simulated robot? A comparison of empathy toward a physical and a simulated robot, in Paper presented at the 2015 10th ACM/IEEE Int. Conf. Human-Robot Interaction (HRI) (Portland, OR, 2015), pp. 125–132. Google Scholar
61. M. Sharma, D. Hildebrandt, G. Newman, J. E. Young and R. Eskicioglu, Communicating affect via flight path: Exploring use of the laban effort system for designing affective locomotion paths, in Paper presented at the Proc. 8th ACM/IEEE Int. Conf. Human-Robot Interaction (2013). Crossref, Google Scholar
62. V. K. Sims, M. G. Chin, H. C. Lum, L. Upham-Ellis, T. Ballion and N. C. Lagattuta, Robots’ auditory cues are subject to anthropomorphism, in Paper presented at the Proc. Human Factors and Ergonomics Society Annual Meeting (2009). Google Scholar
63. S. Song and S. Yamada, Expressing emotions through color, sound, and vibration with an appearance-constrained social robot, in Paper presented at the Proc. 2017 ACM/IEEE Int. Conf. Human-Robot Interaction (Vienna, Austria, 2017), pp. 2–11. Crossref, Google Scholar
64. J. M. Susskind, D. H. Lee, A. Cusi, R. Feiman, W. Grabski and A. K. Anderson, Expressing fear enhances sensory acquisition, Nat. Neurosci. 11(7) (2008) 843. Crossref, Web of Science, Google Scholar
65. M. L. Walters, D. S. Syrdal, K. L. Koay, K. Dautenhahn and R. Te Boekhorst, Human approach distances to a mechanical-looking robot with different robot voice styles, in Paper presented at the RO-MAN 2008-The 17th IEEE Int. Symp. Robot and Human Interactive Communication (Munich, Germany, 2008), pp. 707–712. Crossref, Google Scholar
66. A. Waytz, J. Heafner and N. Epley, The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle, J. Exp. Soc. Psychol. 52 (2014) 113–117. Crossref, Web of Science, Google Scholar
67. J. M. G. Williams, F. N. Watts, C. MacLeod and A. Mathews, Cognitive Psychology and Emotional Disorders (John Wiley & Sons, New York, 1988). Google Scholar
68. J. O. Wobbrock, L. Findlater, D. Gergle and J. J. Higgins, The aligned rank transform for nonparametric factorial analyses using only ANOVA procedures, in Paper presented at the Proc. SIGCHI Conf. Human Factors in Computing Systems (Vancouver, BC, Canada, 2011), pp. 143–146. Crossref, Google Scholar

Vol. 20, No. 01

Metrics

Downloaded 655 times

History

Received 19 March 2022

Revised 1 December 2022

Accepted 19 December 2022

Published: 22 February 2023

Information

This is an Open Access article published by World Scientific Publishing Company. It is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 (CC BY-NC-ND) License which permits use, distribution and reproduction, provided that the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.

Keywords

PDF download

System Upgrade on Tue, May 28th, 2024 at 2am (EDT)

The Effects of Robot Voices and Appearances on Users’ Emotion Recognition and Subjective Perception

Abstract

Recommended