Daneomatic

My final semester of graduate school is now long over, I have spent the last few weeks immersed in the awesome culture that is Adaptive Path, and yet embodied interaction continues to dominate my thoughts.

Today I have been reviewing my notes from the Hans and Umbach project, after using a terminal command to combine hundreds of text documents into a single file, which I converted to a PDF and loaded onto an iPad that I am currently borrowing from work. One thing that is striking, and perhaps disappointing, is that I originally set out on the modest goal of learning Arduino, hardware sketching and physical computing. Where others have made LED matrixes that let you play Super Mario Brothers, or tanks that they can control with an iPhone, I made a few LED mixers, interfaced with a 7-segment display, took apart a Super Nintendo controller, and played Pong with a Wii Nunchuk.

My results didn’t quite meet my initial expectations. Electronics, it turns out, is still an archaic craft wrapped in cloaks of obtuse language and user-hostile encodings, and is certainly an art unto itself. I realized that to produce the robust interactions I had intended, with all the nuance and detail with which I approach my screen-mediated design work, would take an entire career worth of learning and refinement.

So then, were my efforts with Hans and Umbach all for naught? I don’t believe so. Physical computing exists at the intersection of the physical and the so-called digital worlds, which is why I was originally so interested in studying it. In reflecting extensively on my own process of learning electronics, and simultaneously diving deep on academic research behind notions of embodiment, I came to realize that perhaps stumbling through the craft of linking these two worlds together wasn’t the best use of my strengths.

Because, I realized, the boundary between the physical and digital worlds was a false one.

Indeed, mentally compartmentalizing the physical from the digital makes sense from a computer science perspective, or from a system architecture perspective, but it is a wrong, dead-wrong approach for an interaction design perspective.

Every interaction, whether it is with a coffee cup or a keyboard or pixels on a screen, exists in the physical world, is perceived through our senses, is actively interpreted by us, and is thus rendered meaningful by our interpretation. Whether it is physical or digital, every interaction is embodied, as we only interact with the physical manifestations of digital information.

This was a surprising conclusion to reach, as the whole reason I set out on this inquiry was to prove that the interactions afforded by the devices at the Musee Mecanique were of a different class than those afforded by screens and input devices. What I began to discover, though, is that even our most familiar, most natural, most culturally-embedded interactions, are all technologically-mediated.

There is nothing natural about plain paper, dark ink or the printing press; these are all technologies. However, a book differs greatly from an e-book in terms of the richness of its physicality. Screens typically comprise an interaction that is physically impoverished, given the rich range of sensing capabilities we have as human beings. By not engaging our senses for texture, warmth, smell and sound, the e-book is limited in how it engages our sense of embodiment, but it is embodied nevertheless.

Indeed, too much effort has been wasted trying to explain how and why tangible computing is new and different than what came before it… what, intangible computing? I believe that the assertion is irrelevant, that tangible computing is not new, but as an area of inquiry it has given us a new perspective from which to reconsider all interactions, namely that of their embodied qualities. While tangible computing is mostly concerned with the sense of touch and physical manipulability, embodied interaction considers the larger notions of physicality as a whole, the human body as a mediator of experience, the nature of being, and the role of individual interpretation as central to the formation of meaning.

All interactions can benefit from an embodied perspective, not just analog, physical, in-the-world interactions, but so-called digital ones as well. There are all these things in the world, hardly perceptible but nonetheless important, that we use every day to create meaning.

What I continue to outline, through my consideration of embodied interaction, phenomenology and metaphor, is a means by which we can talk about these experiences in such a way that embodiment can better inform our design process.

I’m not there yet, but I’m continuing to work it out.

And now, I get to work it out with some of the coolest people in the whole entire world.

Okay. It’s been awhile. I’ve taken some time off. From this, as well as from embodied interaction.

But let’s get back at it. Embodied interaction, that is.

I’ve had quite some time to decompress about this, take a pause and see what sort of ideas keep bubbling to the surface. And the results are not surprising. Or are surprising. Or whatever.

On a postmodern interpretation of technology, and rejecting a sense of inevitability.

Jaron Lanier. And questioning underlying assumptions, tacit assumptions, the colorless, odorless nature of our technological surroundings. Of our environments. Render explicit to consciousness that our ecologies are not inevitable, that they are not natural, that they are not predestined, that they are constructed and hold no truths. That is not to say they are completely relative, but just that they are subjective. Indeed. Privacy on the internet. That you cannot hear people from your car. Certain responsibilities, the lines we draw between designer and user, producer and consumer, etc.

Who is responsible when my gas pedal sticks? How about when I swerve to miss a deer? Or if I hit the deer? Should the car allow me to swerve so strong that I can flip it? Should it never be allowed to exceed 55 mph? What is the logical limit for the top speed? What is safe? What is safe enough? Given infinite resources, given no technological constraints, where would we find ourselves?

On the future of screen-mediated interactions.

Screens, for instance. There’s nothing inherently wrong with screens. A screen is just a collection of pixels. RGB emissive light, photons in this case. However, they could just as well be CMYK dots, like a newspaper. Like a magazine. Imagine how that would change our relationship with paper. Indeed, in that case you could pinch-zoom the 2010 Rand McNally U.S. Road Atlas. You’ve already tried that. Jake’s already tried to slide-unlock his wallet.

There is a feedback loop here. Digital technology influences how we interact with our analog environments. Not just vice versa. Twenty years ago the analog interaction of operating a Rolodex offered a logical metaphor for the digital interaction of browsing an address book. But no more. How many 15-year-olds today do you think have operated a Rolodex? How many do you think have operated an iPod? Or an iPhone? One metaphor for interaction here can no longer effectively be leveraged. Methinks the metaphor of the Rolodex interaction is dead.

So, you take Iron Man 2, with its transparent glass screens, and you think, man, that looks cool at first, but then you realize trying to focus on flat content projected on a transparent screen would be somewhat straining. But. If you can project an image on clear glass like that, who is to say you cannot also project a black background? And now, your office may consist of hundreds of screens, but when they’re off they’re transparent. Open. Airy. They barely exist. And the fact that it can be transparent, or can project its own black background for familiar contrast, or yes, that they can be transparent, opens up all sorts of options for augmented reality.

Though, for a truly transparent glass screen (which is merely transitory technology on the way to in-air displays) you come up against problems of auto-stereoscopy and determining the relativistic perspective of the user’s viewpoint… parallax and the like. I move my head, and the display needs to update accordingly. Or, what if I’m sharing the same augmented reality screen with someone else? They need to see a different display, from a different angle, than myself. Here we need some sort of holography, that projects a unique display along all emissive points.

When we think of screens, we need to think beyond the current technological implementation of the screen, and instead think of the screen metaphorically. What are the terms we use to construct our thinking of this display? Can we touch it? Do we touch it? If we touch it does it get greasy? Not necessarily. I predict in a few years that nanotechnology will provide us with materials, perhaps inspired by the leaves of the lotus, that collect no dust, accept no grease (even from the infinitely-greasy human hand). Imagine if all glass were made of such a material.

Is a screen an extension of a book? A viewport into another world? A wormhole? How we align ourselves, socially and culturally, with these artifacts greatly influences how we perceive them, how we conceptualize them, how we imagine ourselves using them. We look back at old science fiction movies and laugh at their cornball conceptualization of the future, but it’s important to recognize that every piece of science fiction is a product of a unique society and culture. Especially mainstream science fiction (or depictions of the future, as seen in Iron Man) needs to consider their sociocultural situatedness.

Ironically, technology in science fiction needs to appear futuristic, but not so much so that it seems unbelievable and unachievable given current understandings of the world. I recently read that plants may use quantum entanglement to maximize the efficiency of photosynthesis, and that quantum entanglement may allow birds to “see” the Earth’s magnetic field, aiding in migration. If these theories turn out to hold weight and thusly become popularized, they will influence our shared, intersubjective world, and become a resource that science fiction can leverage for believably futuristic renderings of the, well, future.

On questioning hegemonies.

I am realizing that one of my roles as a designer is to question, or at least render explicit, the tacit assumptions of the hegemonies in which we conduct our lives. As interaction designers, we have inherited the legacy, a powerful and important legacy at that, of a scientific approach to computation, as well as an initially cognitive-systems approach to interaction. The scientific, non-humanistic origins of our field, I believe, continue to silently influence the way we think about and talk about interaction.

There is a strong, increasingly strong, reaction against these rational histories of human-computer interaction, towards a more experiential model that considers the whole person, their emotions, desires, goals and fears, not only as something to design for, but something to design with. The user as a medium for design. Indeed, the interpretative abilities of the user are an incredible resource that can, nay must, be effectively leveraged by our designs.

The value in a design is not objectively measurable, and is not contained in the designed artifact itself, but in the union between the artifact and the user. The simplest designs are compelling not merely because they are simple, but because they so gracefully leverage the rich intersubjective world of the user (or users) to give them meaning. As phenomenology tells us, these meanings are situated not in the artifact, but in the consciousness of the user herself. Interaction design is concerned not with the objective world, but the messy, subjective world of interpretation. Phenomenology is at the very core of interaction design; concerned with reality as it is revealed to and manifest in consciousness.

I am proud that interaction design is increasing concerned with the messy subjective world, that it realizes that an account of the objective qualities of the world are insufficient to design compelling interactions. Nevertheless, I believe there is still significant work to be done in shrugging off the scientific cloak of computation, so that we can truly design future-facing interactions. I believe certain metaphors used for describing our systems have hung on past their prime, and silently and insidiously damage progress in our field. Most notably, as I have described recently, is the conceptualization of a virtual world that exists independently of the physical world.

On dispelling the myth of the virtual world.

While the difference between the physical and digital is certainly important from a technology and computation perspective, I believe it is meaningless from an interactive perspective. Nevertheless, we still speak of making virtual friends, roaming virtual worlds, or downloading digital information. I believe this categorization creates a false boundary between the physical and digital worlds, mischaracterizing the digital and trivializing the real, physical, embodied interactions that happen, that must happen, when a user interacts with the so-called virtual world.

Interacting with a friend in World of Warcraft is greatly different than interacting with them when they’re standing in your living room, but not because one is a “virtual” interaction and the other is a “real” interaction. No, they are both physical interactions, one mediated in co-present physical space (with all the available expressive faculties that come along with such co-presence), and one mediated through keyboard, mouse, screen and audio. To characterize the latter as “virtual” is to casually dismiss the embodied interactions that must happen in order for the conversation to take place, and to neglect possible opportunities to make the interaction more richly embodied.

On disentangling interaction design from its computational roots.

Computer science must necessarily distinguish between hardware and software layers, either of which can branch into any multitude of sub-disciplines. However, users do not necessarily make any such distinction. I have observed college freshmen working with computers, and their conceptual model of computers often does not distinguish between operating system and application, or even local (as in, on their computer) or remote (as in, on the internet). To them, a computer (or even computation as a whole) is one amorphous interactive mass, which whether we like it or not, is how we have to design it.

Also. We must design in the abstract, but ultimately our design are interacted with at the ultimate particular level. People never abstractly interact with a product. They only particularly, specifically, interact with something.

Or something.

Through their research, Hans and Umbach have discovered that there is no shortage of brilliant work summarizing the primary concepts of embodied interaction. From Antle to Schiphorst, from Dourish to Hornecker, from Robertson to Sharlin to Lowgren to Fernaeus to Djajadiningrat to Fishkin, everyone seems to be reading the right stuff. Everyone is talking about Heidegger and his hermeneutical phenomenology, a philosophical approach to understanding the way the world is manifest in consciousness, how we interpret our experience with the world, and ultimately how we form meanings with it.

Everyone is channeling Dourish, and his work unifying social computing and tangible computing under the banner of embodied interaction. Many authors are channeling Lakoff and Johnson, and their profound work studying linguistics, metaphors and embodied cognition. Indeed, any text that discusses embodied interaction, without reference to Lakoff and Johnson, is immediately suspect in the boys’ book.

Lakoff and Johnson, and the role of metaphor in human thinking.

Lakoff and Johnson posit that much of our language, and thus much of our thinking, is dependent on our use of metaphors to describe the world. These metaphors are so ingrained in our thinking that we are rarely conscious of their use. For example, we describe time using spatial metaphors, or even material metaphors. Things that are in the future are “ahead” of us, and things in the past are “behind” us. We talk about the speed at which we perceive time passing, and we describe time as though it is water, a continually flowing substance. Time slips through our fingers, we don’t have enough of it, and we frequently run out of it.

Lakoff and Johnson argue that metaphors are not just convenient linguistic tricks we use that allow us to communicate more efficiently with one another, but that our brains are hard-wired to categorize and associate in such a way that we can’t help but think in metaphor. Hans and Umbach have definitely experienced that in the last few months, as they’ve been learning electronics. As they work with circuits and components, trying to build things that work and debug things that don’t, they’re constantly using spatial and material metaphors as a foundation to their thinking. We talk about electricity “flowing” from negative to positive, as though it is water. We talk about resistors resisting (or constricting) the flow of electricity. We talk about capacitors “filling up”, or buttons “closing” a circuit, or transistors “waiting” for a signal.

If we pause just for a moment, none of these thoughts regarding electricity make any sense at all. We can’t see it, so it’s meaningless to “know” or even to “think” that it acts like water, even while this particular mental model sets us up for success when creating a functional circuit. I close things in my environment all the time, such as doors, windows and notebooks, but to say that a pressed button “closes” a circuit is nonsense. Worst of all, how can a transistor “wait”? For something to wait implies that it perceives time, that it can anticipate the future, that it will respond in some manner when the appropriate stimuli presents itself.

Animals wait. Humans wait. Transistors do not wait, and yet this metaphor, that of the transistor as an organism that can anticipate and respond, tells us how to work with them. This, then, is where Lakoff and Johnson’s work gets particularly juicy. Humans are biological creatures with particular sensory capacities. We see light across a particular spectrum, can sense heat across a particular range at varying degrees of sensitivity, and have bodies with arms, fingers and hands that grant us certain abilities for interacting with the world.

Cognition is situated in the the body, and the body influences cognition.

J.J. Gibson’s work in ecological psychology argues that action and cognition are radically situated in the environment and inseparable from it, such that you can make no predictions about an organism’s behavior without knowing about the environment in which it is situated. Lakoff and Johnson extend Gibson’s work by channeling the concept of embodied cognition, which similarly claims that cognition is radically situated in the body.

Indeed, according to embodied cognition, the reason we perceive the world the way we do is not necessarily because the world possesses certain perceptible qualities, but rather because our bodies perceive and make sense of the world in a certain way. We perceive time in a certain way because we are hard-wired to experience it in that way. We organize the physical world in time because it is impossible for us to organize it independent of time. The more we learn about quantum mechanics, too, the more we learn that there is little in the world that objectively reflects the common sense human experience of time.

This is not to say that the objective world does not exist, but rather that we need to deliberately consider the way our minds make sense of the world. Since our minds are situated in our bodies, and our bodies have certain capabilities that pre-filter our access to the world, the importance of considering subjective experience as a phenomenon independent of the objective world cannot be understated.

“I can’t get my body out of my mind.”

The notion of embodied cognition has profound implications, and we can see some of them manifested in the way we talk about, and orient ourselves towards, the physical world. Our bodies are basically symmetrical from left to right, but strongly asymmetrical from front to back. We can see things when they are in front of us, but not when they are behind us. Our limbs are oriented in such a way that we walk in a forward vector, towards our line of sight.

Thus, things that we encounter “in the future” we typically encounter as we walk towards them, and things that we encountered “in the past” are things that are behind us. This asymmetry from front to back gives rise not only to the way we orient ourselves spatially, but also influences how we perceive the world. In this way our bodies’ unique configuration determines our understanding of time, spatially situating our temporal metaphors.

The richer notions of embodiment that Hans and Umbach have discovered over the course of our project consider these notions of metaphor as a fundamental part of how we interpret the world and make meaning of it. These metaphors arise out of the unique qualities and perceptual capabilities of our bodies, such that the way we make sense of and interact with the world is necessarily shaped by our own physical characteristics.