Home Sweet Smart Home: Discussing the feasibility of smart objects at home

A fridge that automatically re-orders groceries when they are running out, a smart diaper that sends a tweet when your baby gets wet and a voice controlled Barbie’s Smart Home that takes playing with Barbie’s to a whole new level (Bohn; Katz; Mynihan). The business of smart objects is developing fast. How can you keep up with the possibilities and be aware of how the objects work and affect the domestic environment? Home Sweet Smart Home has been created to tackle these questions.

The everyday objects are examples of what Miorandi et al. and Bohn et al. describe as “Internet of Things” (IoT)- objects: objects that contain a microcomputer equipped with various sensors and some kind of communication technology (Miorandi et al. 1497; Bohn et al. 7). As a starting point for discussion around the topic of IoT, we chose the commercial below since it shows in a slightly exaggerating way how the Internet of Things might affect our homes and lives.

The general purpose of the IoT is described by the overarching phenomenon “Ambient Intelligence”, which is concerned with “user-friendliness, user-empowerment and support for human interactions” (Friedewald et al. 222). Although these objects are generally designed to make our lives easier, they also gather and send information to various databases. Bohn et. al. argue that these smart objects play an increasingly important role in our daily lives and have the potential to transform our society (6). These consequences ask for an investigation into what smart objects can do and in what ways the IoT works. This information should, in our opinion, be shared with users to keep them well informed on the latest developments. This is where Home Sweet Smart Home comes in.

Home Sweet Smart Home is a concept for a landing page that aims to inform users of the implications of the data collected from the use of smart domestic objects. Rather than just telling users what the objects can do, this project is concerned with the kind of data that is collected, interpreted and potentially used to encroach upon the sphere of family/domestic life. The aim of the project is to make users think about the question of why and to what extent data from these objects need to be shared and with whom. Does information on your sleeping pattern need to travel further than the infrastructure of your home, or even the smart bed itself? In the next section, we will relate these questions to the current academic debate around the IoT.

Academic Debate

Home Sweet Smart Home pays particular attention to academic debates surrounding privacy, and especially the notion of “dataveillance” that arises from the ability of these smart objects to register many of our interactions in domestic and private spaces. Friedewald et al. mention four functions for what smart objects can be used for in the domestic sphere (225). Our project focuses on the home automation functions and recreational functions that these objects supposedly support.

Throughout our project we use the definition of the Internet of Things as formulated by Atzori et al.:

The basic idea of this concept is the pervasive presence around us of a variety of things or objects – such as Radio-Frequency IDentification (RFID) tags, sensors, actuators, mobile phones, etc. – which, through unique addressing schemes, are able to interact with each other and cooperate with their neighbors to reach common goals (1).

IoT objects can enhance the home experience and make certain tasks easier. But, the objects face some challenges when it comes to integration into the households. Friedewald et al. name several challenges, specifically: “how to make technology learn about the people and their identity: habits, preferences, behavioral patterns, etc. and how to apply such knowledge in varying contexts” (236). Another main point is the issue of privacy and the possible misuse of the data. “The more complex the systems become, the more vulnerable they are to malfunction” (Friedewald 236). This complexity can refer to both the amount of objects that are connected, but also to the actors that are involved in the collecting and sending of data.

Dataveillance

A relevant term in relation to the complexity and dangers of IoT objects is dataveillance. Where Bohn et al. use the notion of surveillance to point out that the smart objects together can turn into a network that can register every move within the house, new problems arise when the data “leaves” the domestic sphere (14). José van Dijck defines the term dataveillance in relation to surveillance: “Dataveillance – the monitoring of citizens on the basis of their online data – differs from surveillance on at least one important account: whereas surveillance presumes monitoring for specific purposes, dataveillance entails the continuous tracking of (meta)data for unstated preset purposes” (205). This continuous tracking of data from the smart objects creates an invisible data infrastructure that users are possibly not aware of.

Feedback Loops

To look at it from a different perspective, we were questioning what companies can do with the data that is collected from the smart objects in households. Karami et al. point out the use of user feedback: “The user feedback can be collected from facial expressions, gestures, vocal expressions, etc.” (Karami et al.). They use the term “feedback loop” to describe the process of improving the objects to suit users needs (Karami et al.).

To make this concept of the feedback loop more tangible, we’ve created a simplified version of how the data flows between several actors. As the image below shows, the smart objects are connected not only to each other but also, possibly, to the smartphones of the users (via apps and Wi-Fi), the manufacturers of the objects and the possible suppliers of products (for example to automatically re-order items from a supplier).

Questions of responsibility

The main purpose of this project is to raise awareness regarding IoT objects. A notion that is closely related to our project is privacy. What this means in relation to IoT is explained by Perera et al. They address privacy concerns with regards to IoT technologies and devices:

In the IoT era, the amount of user data that can be collected will be significantly higher than in the past. For example, recent wearable technologies such as Google Glass, Apple iWatch, Google Fit, Apple Health Kit, and Apple Home Kit can collect sensitive information about users ranging from their health conditions to financial status by observing or recording their daily activities (33).

These concerns succinctly exemplify why IoT technologies intervene in debates over privacy.
We argue that it is important to acknowledge the sensitivity of the information gathered from smart objects and that the purposes for what this information will be used for should be carefully considered. However, where the responsibility lies for the collection, processing and using of this data is not so straightforward.

Van Dijck addresses this need to look into the responsibilities of the parties that are involved in the distribution and use of data. She poses the question: “What kind of critical attitude is required in the face of this complex system of online information flows?” (Van Dijck 198). In the visualisation above, Perera et al. show the stakeholders that they see as the responsible parties in the issue of protecting user privacy (36). Their scheme shows that the individual consumers are also involved, and this is where Home Sweet Smart Home intervenes.

Making the invisible visible

As previously mentioned, Home Sweet Smart Home engages with data infrastructures and also the topic of feedback loops. The invisible infrastructures within the home form the structure of a larger global infrastructure. This overarching infrastructure allows the existence of certain feedback loops- information gathered from user interaction with smart objects being sent to and from various databases and companies for collation and evaluation.

This process of collecting data from different objects and sending it to companies who evaluate this to improve their products is an invisible process which presents various problems. The image shown above visualises how smart objects are connected to various actors, including smartphones, suppliers and manufacturers.

The most pressing of these concerns is that we cannot be sure of what types of data are gathered, or even what data is taken into account when feedback is being generated. There is a grey area between what data is collected and what data is actually used to improve the functioning of these smart objects. Does the data in this grey area get sold on to third parties? Our purpose with Home Sweet Smart Home is to make users who are interested in or already use smart objects aware of such questions. The landing page will provide further information on the objects, especially on the data infrastructure and the quantity and variety of data that is possibly gathered and reused.

The image below is a heat map which is generated based on the number of sensors that are included in the 5 IoT-objects in the landing page. Whereas a smaller blue circle represents few sensors, a bigger red circle represents more sensors.

Concept & Realisation

With Home Sweet Smart Home we developed a drag and drop interface that lets users furnish their homes with smart objects. First, the user can draw his or her apartment and make it as real as possible. Even though the user draws in 2D, the home comes to life through 3D visualisation which lets the user walk through the virtual home.

Home Sweet Smart Home is up to date with IoT home furnishings. The user can choose from a list and place the items in his/her virtual home. Additional information on the IoT objects is gathered and presented to the user using easily accessible language and in an understandable way. The object in question also shows what kind of data it potentially collects. This should encourage the user to think critically about the objects. Eventually, the main goal is to raise awareness with users, potential buyers, and other people who are interested in smart objects.

The drag and drop furnishing tool was inspired by the app Roomle (Roomle.com). It was used to build the basic outline of the example home in our mock ups. To simulate the drag and drop functionality, animation software Hype was used (Tumult) in combination with Photoshop CC (Adobe Systems Software Ireland Ltd., ‘Adobe Photoshop CC’). The 3D walkthrough animation is created using Photoshop CC and After Effects CC (Adobe Systems Software Ireland Ltd., ‘Adobe After Effects CC’). The heatmap animation is created with Photoshop CC and Hype. For the realisation of the the appendix, content management system WordPress was used. The website is visualised through a mock up built in keynote. Further information, literature and our progress of work is documented on this website.

Findings

This project uses the Internet of Things concept as a point of intervention into the debate surrounding privacy and data collection, while also letting us understand the rationale behind smart objects. Besides acting as an informant, we want the realised landing page Home Sweet Smart Home to visualise the use of smart objects and their place within an overall hidden infrastructure.

During the realisation of the Home Sweet Smart Home project, it became clear that the companies producing IoT objects are restrained on providing information about the data collection that is done by those objects, and where the data is going. In user manuals and on product websites, they do mention part of the sensors that are used in the objects. In many cases however, reverse engineering was the only way to find out which sensors could (probably) be found in the objects.

Home Sweet Smart Home is designed to enable users to think critically about IoT objects. By providing information on which data is collected by the objects, and what could be done with this data we try to raise awareness about the possible implication of the IoT. We hope that this will lead to a wider understanding of smart objects and invisible data infrastructures.

www.homesweetsmarthomeblog.wordpress.com

Bibliography

Adobe Systems Software Ireland Ltd. ‘Adobe After Effects CC’. Adobe After Effects CC. <http://www.adobe.com/nl/products/aftereffects.htmlsdid=KKQOW&kw=semgeneric&skwcid=AL!3085!3!86856661824!e!!g!!after%20effects&mv=search&s_kwcid=AL!3085!3!86856661824!e!!g!!after%20effects&ef_id=V1XxAQAABRneZB4D:20161014123923:s>.

—. ‘Adobe Photoshop CC’. Adobe Photoshop CC. <http://www.adobe.com/nl/products/photoshop.html?mv=search&s_kwcid=AL!3085!3!98193109344!b!!g!!adobe%20photoshop%20cs%20download&ef_id=V1XxAQAABRneZB4D:20161014123524:s>.

Atzori, Luigi, Antonio Iera, and Giacomo Morabito. ‘The Internet of Things: A Survey’. Computer Networks 54.15 (2010): 2787–2805.

Bohn, Dieter. ‘Samsung’s New Fridge Can Order Fresh Direct Groceries from Its Humongous Touchscreen – The Verge’. The Verge. 22 Sept. 2016. <http://www.theverge.com/2016/1/5/10708380/samsung-family-hub-fridge-mastercard-app-groceries-ces-2016>.

Bohn, Jürgen et al. ‘Social, Economic, and Ethical Implications of Ambient Intelligence and Ubiquitous Computing’. Ambient Intelligence. Springer, 2005. 5–29. <http://link.springer.com/chapter/10.1007/3-540-27139-2_2>.

Centraal Beheer. ‘Commercial Welkom (2015) – Even Apeldoorn Bellen – Centraal Beheer.’ YouTube. 30 April 2015. 17 October 2016. <https://www.youtube.com/watch?v=_CQA3X-qNgA>.

Dijck, José van. ‘Datafication, dataism and dataveillance: Big Data between scientific paradigm and ideology.’ Surveillance & Society 12.2 (2014): 197-208.

Friedewald, Michael et al. ‘Perspectives of Ambient Intelligence in the Home Environment’. Telematics and Informatics 22.3 (2005): 221–238.

‘GameSalad – Drag & Drop Programming – No Coding Required. Game Creation for Everyone’. Drag & Drop Programming. 25 Sept. 2016. <http://gamesalad.com/>.

Gubbi, Jayavardhana et al. ‘Internet of Things (IoT): A Vision, Architectural Elements, and Future Directions’. Future Generation Computer Systems 29.7 (2013): 1645–1660.

Katz, Leslie. ‘TweetPee: Huggies Sends a Tweet When Baby’s Wet’. CNET. 22 Sept. 2016. <https://www.cnet.com/news/tweetpee-huggies-sends-a-tweet-when-babys-wet/>.

Miorandi, Daniele et al. ‘Internet of Things: Vision, Applications and Research Challenges’. Ad Hoc Networks 10.7 (2012): 1497–1516.

Mynihan, Tim. ‘Barbie’s Hello Dreamhouse Is Better Than Your Smart Home | WIRED’. Barbie’s New Smart Home Is Crushing It So Hard. 22 Sept. 2016. <https://www.wired.com/2016/09/barbies-new-smart-home-crushing-hard/>.

Perera, Charith et al. ‘Big Data Privacy in the Internet of Things Era’. IT Professional 17.3 (2015): 32–39.
Tumult. ‘Tumult Hype’. Tumult Hype 3.5 Create. Animate. HTML5. 14 Oct. 2016. <http://tumult.com/hype/>.

Rianne Riemens (rianne_riemens@hotmail.com)
Marlene Scherf (marlene@marlenescherf.com)
Harry Sullivan (hrsullivan@hotmail.com)
Pieter Vliegenthart (pietervliegenthart@gmail.com)