Book Review ‘What You See Is What You Feel’ by Koert van Mensvoort

In our age of computer mediated images the distinction between reality and the virtual seems to be more blurred than ever. The white beach with palm trees, coconuts and turquoise ocean is no longer that white beach, it has been mediated to a bounty beach. The images we have seen in TV-commercials are mediated into reality. But what experience is more real? Is the experience of a bounty beach, instead of a genuine white beach experience, false or less real? The experience of the user and the fading of reality and vitality is a central theme in the work of Koert van Mensvoort.

During one of my New Media Theory lectures in 2005, when I was still a student at Rotterdam University of applied sciences, the subject was reality and vitality and I got my first Koert van Mensvoort experience. His documentary ‘The woods smell of shampoo‘ got me to think that there is more to it then practical application. Until that time I was only focused on doing my practical work by the book, trying to do it as good as possible. Van Mensvoort was the first theorist that inspired me to look beyond the surface. This resulted in me going to the University of Amsterdam to study Media and Culture, to which I’m – at this moment of writing – a Master Student in New Media. So when I got his new printed book ‘What You See Is What You Feel‘, I had some pretty high expectations, and he didn’t disappoint me at all.

‘What You See Is What You Feel’ is Koert van Mensvoort’s doctoral dissertation, on which he promoted at Eindhoven University of Technology. In this research the distinction between reality and vitality is again his starting point. With his documentary ‘The woods smell of shampoo’ as an example – in which a girl who lives in the city, never visited the woods, and washes her hair with shampoo, that smells like pine tree, visits the woods one day with her father and says “Daddy, the woods smell like shampoo” – Van Mensvoort wants to point out the influence of mediated images in our every day live and discuss the realness of these mediated images. As he puts it; is this girl spoiled by the media or is this child merely fine-tuned herself with the environment she grows up with? According to Van Mensvoort “media technologies are gradually but certainly attaining a level of authority within our society that consequently increases their realness” (van Mensvoort, 2009: p. 14). Van Mensvoort’s research is aimed at, maybe the most influential media technologies of our time; the Graphical User Interface (GUI) .

The GUIs we know nowadays are based on our visual perception, and in some cases combined with sound. Our other senses are not being stimulated while working with a GUI. Taste, smell and touch are not used when getting information from a GUI. Koert van Mensvoort’s research aim is to “contribute to a richer and more physical paradigm of graphical user interfaces” (van Mensvoort, 2009: p. 17). To back this up he demonstrates the importance of touch and the ability to feel in the first chapter of the book. By adding touch he want to enrich the user experience while using this GUI.

This desire to add touch to the computer GUI is not new, mechanical haptic feedback already exists for a while. The most known example of this, is maybe the force feedback joysticks supplied with gaming consoles. When getting shot down in a shoot ’em up game for example, the joystick will vibrate to simulate your death. For the computer, mechanical haptic feedback devices are available but as Van Mensvoort points out in his book they are not widely available, which results in not that many software that is programmed to give force feedback, which results in even less development of new mechanical haptic feedback devices. Because of this problem he aims for a solution without having to introduce any new hardware that is not yet available in every modern home. Another issue he points out is the current GUI we are used to, with its Windows, Icons, Menus and Pointing devices (WIMP). It is so commonly known that it will be hard to introduce any major changes in well known GUIs. The aim of the research is therefore to make use of current WIMPs as much as possible. The result is Optically Simulated Haptic Feedback.

Inspired by renaissance painters – who mathematically and atmospherically enhanced the presents of their work, by using optical tricks – van Mensvoort developed an Optically Simulated Haptic Feedback (OSHF). Seeing the similarities between the canvas and the screens gave Van Mensvoort the idea to apply these optical tricks to the GUI. These optical simulations will give the user a feeling of actual shapes or textures. This is done by active cursor displacement. Holes en Hills are Van Mensvoort’s favorite objects to test it out. When a user tries to hover over a visual hill the cursor is pushed away from it, when the user tries to hover over a visual hole the cursor is pulled in, giving the user the experience of actually hovering over physical hills and holes on the screen.

In chapter three and four of the book Van Mensvoort explains how he  subjected test subjects to different experiments to test his thesis. These chapters are pretty technical, especially if you have no mathematical or programming experience. From my humanities and sociological perspective these chapters where the least interesting. The chapters contain a lot of mathematical formulas and complicated diagrams. The most interesting were the final results of the tests, explaining that the majority of the users preferred Optically Simulated Haptic Feedback over Mechanical Simulated Haptic Feedback and that while using OSHF the test subjects did their work more efficient and with fever mistakes. So OSHF will not only enhance our visual experience but will also help us do our work quicker and with less errors. Chapter three and four are probably required to complete his doctoral dissertation, but if there is ever going to be a second edition of the book I suggest to shorten it to the most imported results. In the meanwhile, readers who are not interested in all the experiments and mathematical formulas, I suggest you skip these chapters.

More interesting on the other hand is Chapter 5 in which Van Mensvoort presents his software toolkit ‘PowerCursor‘. This software toolkit needs to help interaction designers to easily implement the OSHF in their interfaces.  He describes in detail what the toolkit provides and how it can be used. It seems to me a very interesting chapter if you are an interaction designer. With a lot of practical examples, Van Mensvoort gives you enough inspiration on how to use the toolkit and in what cases you can use visual force feedback to optimize your interface. When taking a critical look at this chapter I can only comment that the research for the most suitable technique seems to be out of date. Concluding that flash is the best technique for this toolbox – when researching pretty old techniques like DHTML+Javascript and Director and not exploring more recent techniques like Silverlight and AJAX – is in my opinion a hasty research and conclusion. The outcome of the toolkit however is amazing and a very good first attempt in enriching interfaces with optical haptic feedback.

Even though I’m charmed by Van Mensvoort’s efforts to enrich the user experience of the GUI, I’m somewhat uncertain about the overall though on this research. My personal appreciation for Van Mensvoort work is that he dares to ask the question what is real, but does not take side within this discussion. He point out that the idea of the woods smelling of shampoo can be experienced very real by the city girl. My question arises then why Van Mensvoort is trying to make the GUI more real? What exactly is not real about the current GUI experience that we need to change it? Even though I like the way he manages to add a sense of touch to the GUI, I’m in doubt if it was ever even needed.

My final critic is about the implementation. I’m somewhat skeptical about the results compared to his larger aim of make computer simulation more real. In ‘The Synthetic Image and Its Subject‘ Lev Manovich point us to a property of new media, which exposing it’s technique to the user. Compared to film, in which there is no interactivity between the user and the screen, the picture is experienced more real, according to Manovich. The use of a keyboard or mouse are one of these techniques that are exposed when using new media. By making the mouse cursor more important in haptic feedback I’m not sure if this is an actual step forward or a step backward for the reality experience of new media, or more specific the GUI. According to Lev Manovich these techniques need to disappear to the back in order to make the experience more real. Is making the mouse more important helping to achieve this?

Beside these few critical notes I’m impressed by Koert van Mensvoort’s research. The outcome is interesting and may even be the starting point of a new paradigm of visual interaction design. The book is written in a very understandable way, with lot of images and examples to supports his research. Professional jargon is explained in detail which make is an easy accessible book for less experienced readers of haptic feedback. Overall this book is a must read for every professional interaction designer and a recommendation to everyone who is interested in the future of interaction design.