1. bookVolume 2020 (2020): Issue 2 (April 2020)
Journal Details
First Published
16 Apr 2015
Publication timeframe
4 times per year
access type Open Access

Listen Only When Spoken To: Interpersonal Communication Cues as Smart Speaker Privacy Controls

Published Online: 08 May 2020
Volume & Issue: Volume 2020 (2020) - Issue 2 (April 2020)
Page range: 251 - 270
Received: 31 Aug 2019
Accepted: 16 Dec 2019
Journal Details
First Published
16 Apr 2015
Publication timeframe
4 times per year

Internet of Things and smart home technologies pose challenges for providing effective privacy controls to users, as smart devices lack both traditional screens and input interfaces. We investigate the potential for leveraging interpersonal communication cues as privacy controls in the IoT context, in particular for smart speakers. We propose privacy controls based on two kinds of interpersonal communication cues – gaze direction and voice volume level – that only selectively activate a smart speaker’s microphone or voice recognition when the device is being addressed, in order to avoid constant listening and speech recognition by the smart speaker microphones and reduce false device activation. We implement these privacy controls in a smart speaker prototype and assess their feasibility, usability and user perception in two lab studies. We find that privacy controls based on interpersonal communication cues are practical, do not impair the smart speaker’s functionality, and can be easily used by users to selectively mute the microphone. Based on our findings, we discuss insights regarding the use of interpersonal cues as privacy controls for smart speakers and other IoT devices.


[1] Noura Abdi, Kopo M. Ramokapane, and Jose M. Such. More than smart speakers: Security and privacy perceptions of smart home personal assistants. In Fifteenth Symposium on Usable Privacy and Security (SOUPS 2019), Santa Clara, CA, August 2019. USENIX Association.Search in Google Scholar

[2] Mark Altosaar, Roel Vertegaal, Changuk Sohn, and Daniel Cheng. Auraorb: Using social awareness cues in the design of progressive notification appliances. In Proceedings of the 18th Australia Conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments, OZCHI ’06, pages 159–166, New York, NY, USA, 2006. ACM.Search in Google Scholar

[3] Tawfiq Ammari, Jofish Kaye, Janice Y. Tsai, and Frank Bentley. Music, search, and iot: How people (really) use voice assistants. ACM Trans. Comput.-Hum. Interact., 26(3):17:1–17:28, April 2019.10.1145/3311956Search in Google Scholar

[4] Newgenn Apps. 13 IoT Statistics Defining the Future of Internet of Things. Newgenn Apps, https://www.newgenapps.com/blog/iot-statistics-internet-of-things-future-research-data, 2018. Online; accessed 09/01/2018.Search in Google Scholar

[5] Richard Baguely and Colin McDonald. “Appliance Science: Alexa, how does Alexa work? The science of the Amazon Echo”. CNET, https://www.cnet.com/news/appliance-science-alexa-how-does-alexa-work-the-science-of-amazons-echo/, 2016. Online; accessed 02/26/2019.Search in Google Scholar

[6] Aaron Bangor, Philip Kortum, and James Miller. Determining what individual sus scores mean: Adding an adjective rating scale. J. Usability Studies, 4(3):114–123, May 2009.Search in Google Scholar

[7] Brian Barret. What Amazon Echo and Google Home do with your voice data. Wired, https://www.wired.com/story/amazon-echo-and-google-home-voice-data-delete/, November 2017. Online: accessed 05/08/2018.Search in Google Scholar

[8] John Brooke. SUS – a quick and dirty usability scale. In Usability evaluation in industry. Taylor & Francis, London, 1996.Search in Google Scholar

[9] Joseph Bugeja, Andreas Jacobsson, and Paul Davidsson. On privacy and security challenges in smart connected homes. In Intelligence and Security Informatics Conference (EISIC), pages 172–175. IEEE, 2016.10.1109/EISIC.2016.044Search in Google Scholar

[10] Daniel J Butler, Justin Huang, Franziska Roesner, and Maya Cakmak. The privacy-utility tradeoff for remotely teleoperated robots. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, pages 27–34. ACM, 2015.10.1145/2696454.2696484Search in Google Scholar

[11] Varun Chandrasekaran, Kassem Fawaz, Bilge Mutlu, and Suman Banerjee. Characterizing privacy perceptions of voice assistants: A technology probe study. arXiv preprint, arXiv:1812.00263, 2018.Search in Google Scholar

[12] Goodwin Charles. Conversational organization: Interaction between speakers and hearers. New York, Academic Press, 1981.Search in Google Scholar

[13] Peng Cheng, Ibrahim Ethem Bagci, Jeff Yan, and Utz Roedig. Smart speaker privacy control-acoustic tagging for personal voice assistants. In IEEE Workshop on the Internet of Safe Things (SafeThings 2019), 2019.10.1109/SPW.2019.00035Search in Google Scholar

[14] Eun Kyoung Choe, Sunny Consolvo, Jaeyeon Jung, Beverly Harrison, and Julie A Kientz. Living in a glass house: a survey of private moments in the home. In Proceedings of the 13th international conference on Ubiquitous computing, pages 41–44. ACM, 2011.Search in Google Scholar

[15] Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint, arXiv:1805.10190, 2018.Search in Google Scholar

[16] Ry Crist and Andrew Gebhart. Everything you need to know about the Amazon Echo. CNET, https://www.cnet.com/how-to/amazon-echo-alexa-everything-you-need-to-know/, 2017. Online; accessed 02/26/2019.Search in Google Scholar

[17] Mark L Daly and Knapp John Augustine. Handbook of interpersonal communication. Sage, 2002.Search in Google Scholar

[18] Tamara Denning, Cynthia Matuszek, Karl Koscher, Joshua R Smith, and Tadayoshi Kohno. A spotlight on security and privacy risks with future household robots: attacks and lessons. In Proceedings of the 11th international conference on Ubiquitous computing, pages 105–114. ACM, 2009.10.1145/1620545.1620564Search in Google Scholar

[19] Joseph A DeVito. Interpersonal communication. New York: Longman Inc, 2007.Search in Google Scholar

[20] Nora A Draper and Joseph Turow. The corporate cultivation of digital resignation. New Media & Society, 21(8):1824–1839, 2019.10.1177/1461444819833331Search in Google Scholar

[21] Jide S. Edu, Jose M. Such, and Guillermo Suarez-Tangil. Smart home personal assistants: A security and privacy review. arXiv preprint, arXiv:1903.05593, 2019.Search in Google Scholar

[22] Serge Egelman, Raghudeep Kannavara, and Richard Chow. Is this thing on?: Crowdsourcing privacy indicators for ubiquitous sensing platforms. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 1669–1678. ACM, 2015.10.1145/2702123.2702251Search in Google Scholar

[23] Stefano Fasciani. Voice Controlled interface for Digital Musical Instrument. PhD thesis, 2014. Online; accessed 05/09/2018.Search in Google Scholar

[24] Brian Feldman. “like everyone else, amazon’s alexa is laughing at you”. New York Magazine, http://nymag.com/intelligencer/2018/03/amazon-alexa-is-laughing-atyou.html, 2018. Online; accessed 02/26/2019.Search in Google Scholar

[25] Huan Feng, Kassem Fawaz, and Kang G Shin. Continuous authentication for voice assistants. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking, pages 343–355. ACM, 2017.10.1145/3117811.3117823Search in Google Scholar

[26] Catherine S Fichten, Vicki Tagalakis, Darlene Judd, John Wright, and Rhonda Amsel. Verbal and nonverbal communication cues in daily conversations and dating. The Journal of Social Psychology, 132(6):751–769, 1992.10.1080/00224545.1992.9712105Search in Google Scholar

[27] Nathaniel Fruchter and Ilaria Liccardi. Consumer attitudes towards privacy and security in home assistants. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 2018.10.1145/3170427.3188448Search in Google Scholar

[28] Chuhan Gao, Varun Chandrasekaran, Kassem Fawaz, and Suman Banerjee. Traversing the quagmire that is privacy in your smart home. In Proceedings of the 2018 Workshop on IoT Security and Privacy, pages 22–28. ACM, 2018.Search in Google Scholar

[29] Tom Gross. Ambient interfaces in a web-based theatre of work. In Proceedings of the 10th Euromicro Workshop on Parallel, Distributed and Network-based Processing, pages 55–62. IEEE, 2002.10.1109/EMPDP.2002.994231Search in Google Scholar

[30] Susumu Harada, Jacob O Wobbrock, and James A Landay. Voice games: investigation into the use of non-speech voice input for making computer games more accessible. In IFIP Conference on Human-Computer Interaction, pages 11–29. Springer, 2011.10.1007/978-3-642-23774-4_4Search in Google Scholar

[31] Richard Jones. Communication in the real world: An introduction to communication studies. The Saylor Foundation, 2013.Search in Google Scholar

[32] Bjorn Karmann. Project alias. https://bjoernkarmann.dk/project_alias, 2018. Online; accessed 08/22/2019.Search in Google Scholar

[33] Jacob Kastrenakes. Amazon now lets you tell Alexa to delete your voice recordings. The Verge, https://www.theverge.com/2019/5/29/18644027/amazon-alexa-delete-voice-recordings-command-privacy-hub, 2019. Online; accessed 08/28/2019.Search in Google Scholar

[34] Bastian Könings, Florian Schaub, and Michael Weber. Privacy and trust in ambient intelligent environments. In Next Generation Intelligent Environments, pages 133–164. Springer, 2016.10.1007/978-3-319-23452-6_4Search in Google Scholar

[35] Saadi Lahlou, Marc Langheinrich, and Carsten Röcker. Privacy and trust issues with invisible computers. Commun. ACM, 48(3):59–60, March 2005.10.1145/1047671.1047705Search in Google Scholar

[36] Marc Langheinrich and Florian Schaub. Privacy in mobile and pervasive computing. Synthesis Lectures on Mobile and Pervasive Computing, 10(1):1–139, 2018.10.2200/S00882ED1V01Y201810MPC013Search in Google Scholar

[37] Josephine Lau, Benjamin Zimmerman, and Florian Schaub. Alexa, are you listening?: Privacy perceptions, concerns and privacy-seeking behaviors with smart speakers. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):102, 2018.Search in Google Scholar

[38] Hosub Lee and Alfred Kobsa. Privacy preference modeling and prediction in a simulated campuswide iot environment. In 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 276–285. IEEE, 2017.10.1109/PERCOM.2017.7917874Search in Google Scholar

[39] James R. Lewis. The system usability scale: Past, present, and future. International Journal of Human–Computer Interaction, 34(7):577–590, 2018.Search in Google Scholar

[40] Yuting Liao, Jessica Vitak, Priya Kumar, Michael Zimmer, and Katherine Kritikos. Understanding the role of privacy and trust in intelligent personal assistant adoption. In International Conference on Information, pages 102–113. Springer, 2019.10.1007/978-3-030-15742-5_9Search in Google Scholar

[41] Nathan Malkin, Joe Deatrick, Allen Tong, Primal Wijesekera, Serge Egelman, and David Wagner. Privacy attitudes of smart speaker users. Proceedings on Privacy Enhancing Technologies, 2019(4):250–271, 2019.10.2478/popets-2019-0068Search in Google Scholar

[42] Aadil Mamuji, Roel Vertegaal, J Shell, Thanh Pham, and Changuk Sohn. Auralamp: Contextual speech recognition in an eye contact sensing light appliance. In Extended abstracts of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), 2003.Search in Google Scholar

[43] Zied Mani and Ines Chouk. Drivers of consumers’ resistance to smart products. Journal of Marketing Management, 33(1-2):76–97, 2017.10.1080/0267257X.2016.1245212Search in Google Scholar

[44] Martin, Taylor. 12 ways to use Alexa in the kitchen. CNET, https://www.cnet.com/how-to/amazon-echo-ways-to-usealexa-in-the-kitchen/, November 2017. Online; accessed 05/05/2018.Search in Google Scholar

[45] Evelyn McClave, Helen Kim, Rita Tamer, and Milo Mileff. Head movements in the context of speech in arabic, bulgarian, korean, and african-american vernacular english. Gesture, 7(3):343–390, 2007.10.1075/gest.7.3.04mccSearch in Google Scholar

[46] Emily McReynolds, Sarah Hubbard, Timothy Lau, Aditya Saraf, Maya Cakmak, and Franziska Roesner. Toys that listen: A study of parents, children, and internet-connected toys. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, pages 5197–5207, New York, NY, USA, 2017. ACM.10.1145/3025453.3025735Search in Google Scholar

[47] Pardis Emami Naeini, Sruti Bhagavatula, Hana Habib, Martin Degeling, Lujo Bauer, Lorrie Cranor, and Norman Sadeh. Privacy expectations and preferences in an iot world. In Symposium on Usable Privacy and Security (SOUPS), 2017.Search in Google Scholar

[48] Kayako Nakagawa, Masahiro Shiomi, Kazuhiko Shinozawa, Reo Matsumura, Hiroshi Ishiguro, and Norihiro Hagita. Effect of robot’s whispering behavior on people’s motivation. International Journal of Social Robotics, 5(1):5–16, Jan 2013.10.1007/s12369-012-0141-3Search in Google Scholar

[49] Michael Natale. Convergence of mean vocal intensity in dyadic communication as a function of social desirability. Journal of Personality and Social Psychology, 32(5):790, 1975.10.1037/0022-3514.32.5.790Search in Google Scholar

[50] David H Nguyen and Elizabeth D Mynatt. Privacy mirrors: understanding and shaping socio-technical ubiquitous computing systems. Technical report, Georgia Institute of Technology, 2002.Search in Google Scholar

[51] Carol AE Nickerson. A note on “a concordance correlation coefficient to evaluate reproducibility”. Biometrics, pages 1503–1507, 1997.10.2307/2533516Search in Google Scholar

[52] Katarzyna Olejnik, Italo Dacosta, Joana Soares Machado, Kévin Huguenin, Mohammad Emtiyaz Khan, and Jean-Pierre Hubaux. Smarper: Context-aware and automatic runtime-permissions for mobile devices. In IEEE Symposium on Security and Privacy, pages 1058–1076. IEEE, 2017.10.1109/SP.2017.25Search in Google Scholar

[53] Antti Oulasvirta, Aurora Pihlajamaa, Jukka Perkiö, Debarshi Ray, Taneli Vähäkangas, Tero Hasu, Niklas Vainio, and Petri Myllymäki. Long-term effects of ubiquitous surveillance in the home. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pages 41–50. ACM, 2012.10.1145/2370216.2370224Search in Google Scholar

[54] Richard A. Page and Joseph L. Balloun. The effect of voice volume on the perception of personality. The Journal of Social Psychology, 105(1):65–72, 1978.10.1080/00224545.1978.9924091Search in Google Scholar

[55] Rebecca S Portnoff, Linda N Lee, Serge Egelman, Pratyush Mishra, Derek Leung, and David Wagner. Somebody’s watching me?: Assessing the effectiveness of webcam indicator lights. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 1649–1658. ACM, 2015.10.1145/2702123.2702164Search in Google Scholar

[56] Amanda Purington, Jessie G. Taft, Shruti Sannon, Natalya N. Bazarova, and Samuel Hardman Taylor. “alexa is my new bff”: Social roles, user satisfaction, and person-ification of the amazon echo. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pages 2853–2859, New York, NY, USA, 2017. ACM.10.1145/3027063.3053246Search in Google Scholar

[57] Nirupam Roy, Haitham Hassanieh, and Romit Roy Choudhury. Backdoor: Sounds that a microphone can record, but that humans can’t hear. GetMobile: Mobile Computing and Communications, 21(4):25–29, 2018.Search in Google Scholar

[58] Nirupam Roy, Sheng Shen, Haitham Hassanieh, and Romit Roy Choudhury. Inaudible voice commands: The long-range attack and defense. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’18), pages 547–560, 2018.Search in Google Scholar

[59] Florian Schaub. Context-adaptive privacy mechanisms. In Aris Gkoulalas-Divanis and Claudio Bettini, editors, Handbook of Mobile Data Privacy, pages 337–372. Springer International Publishing, Cham, 2018.10.1007/978-3-319-98161-1_13Search in Google Scholar

[60] Florian Schaub, Rebecca Balebako, Adam L Durity, and Lorrie Faith Cranor. A design space for effective privacy notices. In Eleventh Symposium On Usable Privacy and Security (SOUPS 2015), pages 1–17, 2015.Search in Google Scholar

[61] Florian Schaub, Bastian Könings, and Michael Weber. Context-adaptive privacy: Leveraging context awareness to support privacy decision making. IEEE Pervasive Computing, 14(1):34–43, Jan 2015.10.1109/MPRV.2015.5Search in Google Scholar

[62] Florian Schaub, Bastian Könings, Peter Lang, Björn Wiedersheim, Christian Winkler, and Michael Weber. Prical: context-adaptive privacy in ambient calendar displays. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pages 499–510. ACM, 2014.Search in Google Scholar

[63] Bill Schilit, Norman Adams, and Roy Want. Context-aware computing applications. In First Workshop on Mobile Computing Systems and Applications, pages 85–90. IEEE, 1994.10.1109/WMCSA.1994.16Search in Google Scholar

[64] Albrecht Schmidt. Implicit human computer interaction through context. Personal Technologies, 4(2):191–199, Jun 2000.10.1007/BF01324126Search in Google Scholar

[65] Singleton, Micah. Alexa can now set reminders for you. The Verge, https://www.theverge.com/circuitbreaker/2017/6/1/15724474/alexa-echo-amazon-reminders-named-timers, June 2017. Online; accessed 05/05/2018.Search in Google Scholar

[66] Adam J. Sporka, Sri H. Kurniawan, and Pavel Slavík. Acoustic control of mouse pointer. Universal Access in the Information Society, 4(3):237–245, Mar 2006.10.1007/s10209-005-0010-zSearch in Google Scholar

[67] Blase Ur, Jaeyeon Jung, and Stuart Schechter. Intruders versus intrusiveness: teens’ and parents’ perspectives on home-entryway surveillance. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pages 129–139. ACM, 2014.Search in Google Scholar

[68] Roel Vertegaal, Aadil Mamuji, Changuk Sohn, and Daniel Cheng. Media eyepliances: Using eye tracking for remote control focus selection of appliances. In CHI ’05 Extended Abstracts on Human Factors in Computing Systems, pages 1861–1864, New York, NY, USA, 2005. ACM.10.1145/1056808.1057041Search in Google Scholar

[69] Roel Vertegaal, Robert Slagter, Gerrit van der Veer, and Anton Nijholt. Eye gaze patterns in conversations: There is more to conversational agents than meets the eyes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 301–308, New York, NY, USA, 2001. ACM.10.1145/365024.365119Search in Google Scholar

[70] Mark Weiser. Some computer science issues in ubiquitous computing. Commun. ACM, 36(7):75–84, July 1993.10.1145/159544.159617Search in Google Scholar

[71] Joan Welkowitz, Stanley Feldstein, Mark Finklestein, and Lawrence Aylesworth. Changes in vocal intensity as a function of interspeaker influence. Perceptual and Motor Skills, 35(3):715–718, 1972.10.2466/pms.1972.35.3.715Search in Google Scholar

[72] Primal Wijesekera, Arjun Baokar, Lynn Tsai, Joel Reardon, Serge Egelman, David Wagner, and Konstantin Beznosov. The feasibility of dynamically granted permissions: Aligning mobile privacy with user preferences. In IEEE Symposium on Security and Privacy, pages 1077–1093. IEEE, 2017.10.1109/SP.2017.51Search in Google Scholar

[73] Danny Wyatt, Tanzeem Choudhury, and Henry Kautz. Capturing spontaneous conversation and social dynamics: A privacy-sensitive data collection effort. In IEEE International Conference on Acoustics, Speech and Signal Processing, volume 4, pages IV–213. IEEE, 2007.10.1109/ICASSP.2007.367201Search in Google Scholar

[74] Eric Zeng, Shrirang Mare, and Franziska Roesner. End user security & privacy concerns with smart homes. In Symposium on Usable Privacy and Security (SOUPS), 2017.Search in Google Scholar

[75] Serena Zheng, Marshini Chetty, and Nick Feamster. User perceptions of privacy in smart homes. arXiv preprint, arXiv:1802.08182, 2018.Search in Google Scholar

Recommended articles from Trend MD

Plan your remote conference with Sciendo