Neural Networks: The Technology Behind FaceApp
There are countless photo editing apps available online, with dozens, if not hundreds, letting you change your face through the use of filters. So what is different about FaceApp? Well, this application does not use filters as we know them, but its creators developed a new technology which uses artificial neural networks to edit ‘selfies’ while keeping them photorealistic. What is an artificial neural network?
Artificial Neural Networks
Djekic, Ibric and Primorac define an artificial neural network (ANN) as “an intelligent non-linear mapping system built to loosely simulate the functions of the human brain” (1). It consists of nodes and its connections. Livingstone calls it “a branch of artificial intelligence (AI) research that aims to simulate intelligent behaviour by mimicking the way that biological neural networks work.” (Livingstone v). An artificial neural network is different from most AI methods because it tries to imitate the way in which human intelligence does something, not what it does. The development of artificial neural networks started all the way back in the 1940s, when McCulloch and Pitts created a mathematical model for neurons. The study of ANNs only started developing at a faster pace with the creation of the International Neural Network Society in 1987, with annual meetings and symposiums increasing the knowledge on the topic (Zhang 3-5). In terms of using neural networks in photo editing, this is a much more recent development, only seeing results in the last few years.
FaceApp
FaceApp was created by a small team of Russian developers and released in 2016. To use the app, you can take a ‘selfie’ or use an old photo from your camera roll and apply a filter. The current free filters include a smile, making you look old or young, gender bending, having facial hair, or the classic Walter White look from the TV show Breaking Bad. The app offers additional filters for its paying users, including adding makeup, glasses or different facial hair to your face. Founder and CEO of the developing company Wireless Lab, Yaroslav Goncharov believes that what differentiates his product from other photo editors is the photorealism, which it achieves because the AI analyzes the original image when applying a filter.
Example of the use of FaceApp
In talking to TechCrunch about his app, Goncharov said that they are quite ahead in terms of the technology used, as he didn’t know of any other products or research of similar quality in this area (Lomas, FaceApp uses neural networks for photorealistic selfie tweaks). The team of four engineers developed the technology in-house, but they also used AI open-source libraries like TensorFlow. In terms of the specific technology the developers used, Goncharov said his app uses “deep generative convolutional neural networks”. He continues, in his TechCrunch interview:
‘This technology is quite mature for some tasks, such as artistic style transfer or super-resolution, but has extreme challenges for photo-realistic tasks especially with high resolution images, I don’t think similar effects can be achieved with conventional (not deep learning) algorithms.
Adding a smile may look like a simple modification on the surface but, in fact, is extremely difficult. Smiling is not just lips moving — the entire face changes in a subtle, but complicated, way. Also, for this effect to work on most real life photos, you need to account for too many factors: posture, lighting, skin color, shape of lips and eyes, photo quality, etc.’
(Lomas, FaceApp uses neural networks for photorealistic selfie tweaks)
The results of the editing vary based on the picture used, but some filters look more realistic than others (for example, the smile filter seems to fit much better than the gender-bending filter).
Teens vs. FaceApp (React)
Other Uses of Neural Networks in Photo Editing
Neural Networks have been used in developing other photo editing applications. On a smaller scale than FaceApp, Teleport is using artificial neural networks in order to change your hair or skin colour or the background in a selfie. While the realism of the result depends on the photo and the chosen colour, the use of AI makes the photos look more realistic than other apps or websites that try to do the same thing. When talking about the development of the app, co-founder Victor Koch said “The idea was born out of a set of experiments using neural networks… On complex problems, wide and deep networks significantly outperform small networks and other methods based on manual feature creation due to their flexibility but require a sufficient amount of data to avoid overfitting.” (Lomas, Teleport’s neural networks let you try before you hair dye). Just like the creators of FaceApp, they used TensorFlow to train their models.
Prisma is another app that uses neural networks to change photos, but unlike the first two, it is
Example of the use of Prisma
not made specifically to edit selfies. Prisma edits photos with different artwork or graphic styles. Some of its initial styles offered were a graphic novel style or The Scream by Edvard Munch. In talking to TechCrunch about the development of the app, co-founder Alexey Moiseenkov explains how the app works: neural networks extract the style of the original artwork and apply it to the chosen photo. It’s not a simple filter, the image is created from scratch, combining the two inputs: the style and the picture (Lomas, Prisma uses AI to turn your photos into graphic novel fodder double quick). An app similar to Prisma is Pikazo, which does the same thing, making regular pictures look like artwork or posters, but is slower. Pikazo uses the open-source library Torch (Pangburn).
Researchers at the University of Edinburgh have also created an experimental desktop application called Neural Photo Editor, which can airbrush pictures. Brock et al. describe their creation as “an interface that leverages the power of generative neural networks to make large, semantically coherent changes to existing images” (1). In this app, you use a “contextual paintbrush” and select a colour. The system analyzes it in the context of the photo and produces an intelligent result: painting over someone’s hair can change the hair colour and style. Painting over someone’s mouth with the white brush to create or make a bigger smile. The system is still basic, and so far works best with system-generated photos. While it doesn’t work as well on existing images, the use of “introspective adversarial networks”, a type of deep learning, it can still apply changes to existing photos (Mathies). The Introspective Adversarial Network is a concept created by Brock et al., “a novel hybridization of the VAE (Variational Autoencoder) and GAN (Generative Adversarial Network) “(1). This app is still in its early stages and it requires someone tech savy to be able to use it for now, but its concept could easily turn into the future of photo editing and manipulation.
The use of neural networks in photo editing is clearly still developing and it will still be a while until we can talk about it potentially ‘taking over’ this field, but it is important to know about it and understand it nonetheless. When it comes to apps like FaceApp or the Neural Photo Editor, which can alter someone’s look in a realistic way, their relevance and development feels even more important. In a world where everyone is obsessed with looking their best, and Photoshop is so widely used, neural network-based apps could have a bright future, making it easier for everyone to manipulate photos without the need to understand the process the way you do in Photoshop.
Bibliography
Brock, Andrew et al. Neural Photo Editing with Introspective Adversarial Networks. 2016 https://arxiv.org/abs/1609.07093
Dent, Steve. ‘Photo Editor Uses Neural Networks to Airbrush Like Pro’. Engadget. 2016. 19 September 2017 https://www.engadget.com/2016/09/28/photo-editor-uses-neural-networks-to-airbrush-like-a-pro/
Djekic, Ljiljana, Ibric, Svetlana and Primorac, Marija. ‘Application of Artificial Neural Networks (ANNs) in Development of Pharmaceutical Microemulsions’. Focus on Artificial Neural Networks. Ed. John Flores. New York: Nova Science Publishers. 2011
Graupe, Daniel. Principles of Artificial Neural Networks. 3rd edition. Singapore: World Scientific Publishing. 2013
Livingstone, David, ed. Artificial Neural Networks: Methods and Applications. Sandown: Humana Press. 2008
Lomas, Natasha. ‘FaceApp Uses Neural Networks for Photorealistic Selfie Tweaks’. TechCrunch. 2017. 19 September 2017 https://techcrunch.com/2017/02/08/faceapp-uses-neural-networks-for-photorealistic-selfie-tweaks/
Lomas, Natasha. ‘Prisma Uses AI to Turn Your Photos Into Graphic Novel Fodder Double Quick’. TechCrunch. 2016. 19 September 2017 https://techcrunch.com/2016/06/24/prisma-uses-ai-to-turn-your-photos-into-graphic-novel-fodder-double-quick/
Lomas, Natasha. ‘Teleport’s Neural Networks Let You Try Before You Hair Dye’. TechCrunch. 2017. 19 September 2017 https://techcrunch.com/2017/08/09/teleports-neural-networks-let-you-try-before-you-hair-dye/
Mathies, Daven. ‘FaceApp Uses Neural Networks to Change Your Look, Now Available on Android’. Digital Trends. 2017. 19 September 2017 https://www.digitaltrends.com/photography/faceapp-neural-net-image-editing/
Mathies, Daven. ‘Neural Photo Editor Works Like Magic Thanks to Machine Learning’. Digital Trends. 2016. 19 September 2017 https://www.digitaltrends.com/photography/neural-photo-editor/
Pangburn, DJ. ‘ ‘Pikazo’ App Lets You Paint Neural Network Art Masterpieces’. Vice Creators. 2016. 19 September 2017 https://creators.vice.com/en_us/article/jpvnak/pikazo-neural-network-art-app
Vincent, James. ‘This App Uses Neural Networks to Put a Smile on Anybody’s Face’. The Verge. 2017. 19 September 2017 https://www.theverge.com/tldr/2017/1/27/14412814/faceapp-neural-networks-ai-smile-image-manipulation
Zhang, Wenjun. Computational Ecology: Artificial Neural Networks and Their Applications. Singapore: World Scientific Publishing. 2010
