ARKit: AR Into Your iPhone
With the launch of their new operation system iOS 11 in September 2017, Apple also released its new ARKit tool (Eadiccio). Claiming to be ”the biggest Augmented Reality platform”, Apple now provides software developers to produce an application using Augmented Reality (AR) features (Apple Developer). Augmented Reality systems can combine real and virtual objects in 3D (Azuma 356). It seems like Apple opens doors for user-experiences where real and virtual spaces are combined (Apple Developer). Using the camera of your device, you can for example add 2D or 3D elements to a live view. How quickly users will adapt to these new features is unsure, previous research has shown that the acceptance of AR can be challenging (Olsson and Salo). But now that the broader use of AR applications is accessible, this asks for a better understanding of what the applications will do and the way they do it. Therefore, this text provides a basis. It will be this texts purpose to provide some context for new insights on the use of Augmented Reality both from a developers perspective and an users perspective, regarding privacy challenges.
Apple’s way of creating opportunities
Even though the use AR is not necessarily new, it has never been to developers attention as much as in 2017. Running solely on iOS 11, Apple’s ARKit framework opens up for a wide range of developers with different interests. As Apple’s head of marketing Greg Joswiak explains it in the video placed above himself:
“We’re able to bring it to literally hundreds of millions of people with a software update with existing iPhone’s and iPad’s. It puts us in an incredibly unique position.”
Indeed, this tool puts Apple in a very prominent position, market-wise. Where AR has previously been used in for example military services, surveillance systems and medical education (Azuma 360), it shifts to the interests of other fields. Let say for example the gaming industry, where AR could increase the entertainment industry, by merging online and offline to create a real time experience (Krevelen and Poelman). Pokemon GO is a great example of where AR might have been a factor of success in creating an incredibly popular game. But also in the marketing industry. AR can be a very interesting marketing tool for the retailing business. Swedish furniture giant IKEA launched (together with the release of iOS 11) their new IKEA Place App, which allows users to swipe through the furniture catalog while having their favorite home corners in the background (TechCrunch). These kind of applications might change the way we shop forever. Aiming at enhancing consumer experiences or leisure experiences, Apple has put itself in a very attractive position for software developers.
Besides, Apple is the first platform to release a tool that brings AR to a device we all use in our everyday life. Until this year, the use of AR mostly required a specialized device, for example Google Glass (Roesner, Kohno and Molnar), which previously was considered immobile and unwearable. As Azuma already suggested, AR systems would have to become more portable to be useful. With Apple launching their ARKit tool, AR will indeed become more and more phone-based and intriguing in our personal lives. Having arrived at a point of what might be seen as the start of using AR techniques on a daily basis, the launch of ARKit opens up the room for discussion about users privacy issues.
Getting access and controlling it
To give you the augmented reality experience, an AR application needs some input. In gathering their input, AR apps will go through different steps. First comes the data access (Suman et al.). This can be seen as access to sensitive information like your location, voice, face or surroundings (Azuma). Concerns about companies like Apple or IKEA coming into our private spaces through wearable AR devices have already been expressed in the field of research (Kotsios 169). Sensory input can be the first challenge for new AR application developers. This is to take care of their collected data. Of course this is not a concern that is new to new media usage, but because AR asks for a constant input of sensory access, Roesner, Kohno, and Molnar argue the privacy risks may therefore be larger than for the accustomed apps. It has previously already been argued that these issues are underrepresented in AR research and that it would have to get more attention before users fully accept AR (Azuma et al.).
When using an AR application for the first time, users will be asked to give permission for sensory access. When giving consent for permanent access, users need to realize that most AR applications will require non-stop sensory access. Roesner, Kohno, and Molnar argue it will be a challenge for applications to filter out the access for functionality and separate it from in useful access. But when the app has permanent access to those sensors, maliciously apps can misuse these data. So when we are installing an application using AR, it might be a good idea for us to think twice before we press ‘I agree’, because what AR applications will collect can be as intimate as your voice or home environment. Especially because previous research suggested users of social media (e.g. Facebook) often have trouble with configuring their privacy settings correctly (Yabing et al.). This is especially interesting when using AR apps in public spaces. Because even if the user gave his consent to get sensory access, who says the one standing next to you in that selfie is also okay with that? When using an AR application, users might for example be offered a method to alert bystanders about their capturing activities.
Once applications got their access, they start collecting data. Exploring the technique of AR applications a bit more, it is also interesting to highlight this process of data collection. When users for example use multiple AR apps on one iPhone at the same time, it can be a challenge for iOS to let the right app retrieve the right information. This is where it may be easy for malicious apps to steal data from another app, being active at the same time. When multiple apps are active, one could argue it should be possible for users to know or perhaps even choose which app is getting the focus.
Besides getting access through simultaneously using apps, AR apps can also exchange data themselves. This process can be referred to as the cross-application sharing of data. It allows other parties to get access over the data an application gathers. Application makers can make this data available through the design of their API (Application Programming Interface) (Lomboch and Bechmann). It can be seen as an interface that enables software to communicate to other software. Let’s say you are using the IKEA Place App for example and you desire to share the view of your bedroom including your potential new sofa with your partner. There will probably be tones of options to do so, which will most likely all lead to the use of a social application like Facebook or WhatsApp. In doing so, you allow the AR app to share its output with other applications. This process is commonly used by social media companies. Roesner, Kohno, and Molnar argue that output sharing is also necessary when using AR applications, because otherwise AR systems will lose its functionality. So by launching its tool, Apple faces the challenge of letting iOS deal with the different interests that applications have for gathering data. First, the collection of data input needs to be controlled in a way that misuse is difficult. Second, data output needs to be fairly distributed between multiple AR applications. Researchers have already been thinking about ways to let AR applications handle privacy (Suman et al.).
What does Apple think?
With these challenges for users and developers, it is quite interesting to see how Apple itself thought about these kinds of concerns regarding users privacy. Analysing the platforms App Store Review Guidelines, there appear to be no specific changes since the launch of the ARKit tool last week (Apple). Regarding the privacy topic, the website only included ‘apps that utilize ARKit’ amongst the apps that need users consent for data collection and storage. These data are then not to be used for advertising or other use-based data mining, including by third parties. So it is clear now that our faces, voices and locations won’t be used for any form of advertising, but what can they still be used for then? Facebook for example already shared their data with authorities when asked (Byers and Kelly). Aren’t there supposed to be more specific guidelines when our phones are able to for example track and identify our faces and homes? It is important here for users to realize that the collection of data is becoming more intimate with the use of ARKit.
This text aimed to give some insights into changes that might occur now that Apple introduced ARKit to expand the use of AR applications on our iPhone and iPad devices. Although the launch of ARKit might not create an immediate explosion of AR-focused apps, it still gives app developers a platform where they can reach an incredible amount of everyday users. Previous research has shown it can be a challenge for developers to have successful AR applications (Azuma et al.). This has to do with for example perceived usefulness, enjoyment and satisfaction. Concluding it might be a while before us users fully accept AR applications, it is however now a suitable time to think about the way these application will require always-on sensors that allow them to come more and more into our private space.
Apple Developer: ARKit. 2017. Apple Inc. 22 September 2017. <https://developer.apple.com/arkit/>.
Apple Developer: App Store Review Guidelines. 2017. Apple Inc. 20 September 2017. <https://developer.apple.com/app-store/review/guidelines/>
Azuma, Ronald. ”A Survey of Augmented Reality.” Presence: Teleoperators and Virtual Environments 6.4 (1997): 355-385. doi:10.1162/pres.1922.214.171.1245
Azuma, Ronald, et al. “Recent Advances in Augmented Reality.” IEEE Computer Graphics and Applications 21.6 (2001): 34-47. doi:10.1109/38.963459
Byers, Dylan, and Kelly, Heather. ”Facebooksays it will hand Russian ads over to Congress.” CNN Money. 2017. 22 September 2017. <http://money.cnn.com/2017/09/21/media/mark-zuckerberg-facebook-russian-ads/index.html>
Eadiccio, Lisa. ”Inside Apple’s Plan to Give You a New Way of Experiencing the World”. Time. 2017. 10 September 2017. <http://time.com/4920332/apple-arkit-apps/>.
Kotsios, Andreas. “Privacy in an Augmented Reality.” International Journal of Law and Information Technology 23.2 (2015): 157-185. doi:10.1093/ijlit/eav003
Krevelen, D. van., and Poelman, Ronald. “A Survey of Augmented Reality Technologies, Applications and Limitations.” International Journal of Virtual Reality 9.2 (2010): 1-20. 16 September 2017.
Liu, Yabing, et al. “Analyzing Facebook Privacy Settings: User Expectations vs. Reality.”
Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference. ACM, 2011. < http://dl.acm.org/citation.cfm?doid=2068816.2068823>.
Lundgren, Ingrid. ”IKEA Place, the Retailer’s First ARKit App, Creates Lifelike Pictures of Furniture in Your Home.” TechCrunch. 10 September 2017. <https://techcrunch.com/2017/09/12/ikea-place-the-retailers-first-arkit-app-creates-lifelike-pictures-of-furniture-in-your-home/>.
Olsson, T., and Salo M. ”Online user survey on current mobile augmented reality applications.” Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality, Basel. (2011): 75–84. New York: IEEE. doi:10.1109/ISMAR.2011.6092372
Roesner, Franziska, et al. “Security and Privacy for Augmented Reality Systems.” Communications of the ACM 57.4 (2014): 88-96. doi:10.1145/2580723.2580730
Suman, Jana, et al. “Enabling Fine-Grained Permissions for Augmented Reality Applications with Recognizers.” USENIX Security Symposium. (2013). 18 September 2017. < https://www.usenix.org/system/files/conference/usenixsecurity13/sec13-paper_jana.pdf>. 18 September 2017.
VICE. ”Apple Is Bringing Augmented Reality to Your iPhone.” YouTube. 5 September 2017. 12 September 2017. https://www.youtube.com/watch?v=1k6gUkLYxq0
Pictures by Wired