Detection of image manipulation in an artistic context
The research of multimedia forensics aims to detect and undo digital image manipulations in regard to ethical and legal concerns arising through circulating fake images. However, digital image manipulation also affords artistic and creative expression. Looking at artistic manipulation techniques in digital and analogue photography, I seek to address the most recent detection techniques hence this regard.
Image manipulation: from analogue to digital
With the „official“ birth of photography in 1839 (Manovich 1) a new medium challenged the notion of technology (Daum 17). As photography in the 20th century was predominantly subject to a technological reputation, „not worthy of being considered an art“, artistic movements such as the pictorialism sought to prove the artistic, „subjective“ (Gitelman 5) qualities of this technical medium by focussing on its painterly and neglecting its scientific, „objective“ (Gitelman 5) affordances (Daum 16). Following the principle „As if art had a formula!“ (Daum 49), pictorialists started to interveine into the „recipe-like“ process in the darkroom in various and virtuous ways, what had remarkable effects on the later notion and aesthetics of art and its relation to technology (Jäger 20). Consequently, a branch of art historical research focussed on the process of image manipulation within analogue photography, tracing (differences of) artists’ interventions (Kemp).
Fig.1, showing a photograph of Maria Luberich (1910), was manipulated with painting materials by the „painter photographer“ Frank Eugene. Schmidt’s research gives insights about Eugene’s manipulation techniques that mainly stem from his professional background as a painter and shed another light on the relation between painting and photography.
Around half a century after the invention of the „first working digital computer“ by Zuse in 1936 (Manovich 5, 6), the artistic affordances of a dominantly technological medium revealed a second time:
With this advent, the experimental interaction between photography and painting took another shape – propelled by digital photography and image editing software. The possibility to „save“ a picture at some stage and later return back to some intermediate step and proceed into different directions, „together with the ease of ‚cutting and pasting‘, prompts artists to make pictures that are composites of many different versions of themselves“ – making it difficult to follow retrospectively what sequence of editing steps took place (Elkins 341). For illustrating this, I show a photograph that I manipulated digitally.
Fig. 2 shows an image resulting from a simple algorithm that calculates the absolute color difference of each pixel between two image sources. As the absolute value function has no reverse, given the result image and only one source image, it is not possible to trace back the second source image, as illustrated below.
Image manipulation detection techniques
As we can see, classical algorithmic techniques for tracing back digital manipulations steps fail in even simple cases. But, with the advent of Artificial Intelligence (AI) new research towards this direction takes its first successful steps. The motivation behind this is not to dismantle artistic decisions, but to respond to manipulations „with a malignant goal“ (Thakur et al. 2). While digital image „editing operations have helped enable creative expression“ (Wang et al. 1), one faces ethical and legal issues since „manipulated or tampered images can be used to delude the public, defame a person’s personality and business as well, change political views or affect the criminal investigation“ (Thakur et al. 1). For this, multimedia forensics work on developing techniques that „can authenticate image being uploaded on social media platform as an original image or a doctored image“ (Thakur et al. 3).
Although „Deepfakes“ are a hot topic of debate rising such concerns (Korshunov & Marcel 1), Wang et al. argue that it is still more urgent to look at „subtle […] manipulations“ through „classic image processing techniques.“ (1) For this, they proposed a model that is trained to recognize „faces warped by the Face-aware Liquify tool in Photoshop.“
Their model does not only „outperform human judgements in determining whether images are manipulated“, in many cases it can detect the local area of the „deformation field“ and unwarp images. Still, „the problem of perfectly restoring the original image remains an open challenge“ (Wang et al. 8). Moreover, their model has a “dataset bias“ since all training took place on images that were solely manipulated through Photoshop’s Liquify tool (see: URL 1). While this promises a reliable application solely for this specific tool, Thakur et al. stress the importance „to have a universal image manipulation detection method in the field of multimedia forensics“ suitable for real-world applications, e.g. in social networks (3). Among the biggest challenges for this research field are sparse training data, image compression techniques as the JPEG format, and „the advancement of image manipulation techniques and post-processing methods“ – often developed by forgers trying to hide manipulations (Thakur et al. 3, 9, 11). Although it is a „very difficult“ task to have such general-purpose detection method, researchers find the most recent deep learning models „promising“ for getting there (Thakur et al. 11).
As pictorialists acknowledged and celebrated „that their personal intervention with the negative would be a visible artefact of an otherwise violent act“ (Ostermann 56) – resisting any description through formulas -, this behaves different for artists working with digital image manipulation today. Through the efforts of such early artistic movements the paths for todays receptivity towards experimental image manipulation were rooted (Geimer 70-71, 98).
Whereas the new possibilities of the easy and „tree“-structured (Elkins 341) digital editing process are fruitful for artistic endeavours, they can be fruitful for „malicious“ endeavours as well – for which multimedia forensics research aim for a technique that is able to reproduce the original „authentic“ image (Thakur et al. 4). Whereas for Benjamin „authenticity“ lays outside „technical […] reproducibility“ (3), I want to recall his request to look at the „new functions“ of art emerging through media that afford reproducibility (7). Thinking about pictorialists’ claim of artistic expression that is not fitting any formula, the artistic manipulation of digital photography contrasts the artistic manipulation in analogue photography as the latter affords an interaction with the autonomous dynamics of the chemical substances in use (Geimer 86). Digital editing software “draw[s] upon” (Eyman 52) analogue manipulation techniques, e.g. simulating the film negative, but it has inherently different features (Smythe 84-85) – offering new challenges and opportunities to artists, researchers, forgers, platforms and everyday users. In this hindsight, the digital research methods from multimedia forensics could become integrated not only into future art historical research, but also artistic practices.
Benjamin, Walter. “The Work of Art in the Age of Mechanical Reproduction, 1936.” (1935).
Daum, Patrick. “University and Diversity in European Pictorialism”, in: Ribemont, Francis (Hg.): Impressionist Camera: pictorial photography in Europe, 1888 – 1918, published on the occasion of the exhibition „Impressionist Camera: Pictorial Photography in Europe“, 1888-1918, Saint Louis Art Museum; London 2016.
Elkins, James. “Art history and the criticism of computer-generated images.” Leonardo 27.4 (1994): 335-342.
Eyman, Douglas. Digital rhetoric: Theory, method, practice. University of Michigan Press, 2015.
Geimer, Peter. Inadvertent Images: A History of Photographic Apparitions. University of Chicago Press, 2018.
Gitelman, Lisa. Raw data is an oxymoron. MIT press, 2013.
Jäger, Gottfried. “The art of abstract photography.” (2002).
Kemp, Cornelia. Unikat, Index, Quelle. Erkundungen zum Negativ in Fotografie und Film. Göttingen, 2015.
Korshunov, Pavel, and Sebastien Marcel. “DeepFakes: a New Threat to Face Recognition? Assessment and Detection.” arXiv (2018): arXiv-1812.
Manovich, Lev. “New media: A user’s guide.” Net. Condition (1999).
Schmidt, Marjen: “Die Technik der Manipulation. Die Glasplattennegative von Frank Eugene”, in: Kemp, Cornelia (Hg.): Unikat, Index, Quelle. Erkundungen zum Negativ in Fotografie und Film. Göttingen 2015, 101-116.
Smythe, Luke. “Toward a ‘Wetter’ Photographic Ethos: Liquid Abstract Photographs and the Hubris of Technology.” State of Flux: Aesthetics of Fluid Materials. Dietrich Reimer Verlag, 2017. 75-88.
Ostermann, Mark: “A Photographic Truth”, in: Kemp, Cornelia (Hg.): Unikat, Index, Quelle. Erkundungen zum Negativ in Fotografie und Film, Göttingen 2015, 40-60.
Thakur, Rahul, and Rajesh Rohilla. “Recent Advances in Digital Image Manipulation Detection Techniques: A brief Review.” Forensic Science International (2020): 110311.
Wang, Sheng-Yu, et al. “Detecting photoshopped faces by scripting photoshop.” Proceedings of the IEEE International Conference on Computer Vision. 2019.