tangible-journal-blog

[tangible-journal-blog]

Week 4, Thursday

Our design approach was based on exploring the possibilities and limitations of flora as input and as a sort of inter-media between human action and the sound output. I suggest that one way of referring to our approach is as xeno-design (Schmeer, 2019). Schmeer suggests that while human-centered design serves well in healthcare and interface design, designers lack a deliberate framework for including and centering non-human beings in the design process. Here xeno approaches come in: overlapping with conceptual and speculative design approaches yet explicitelly centering the Other. She proposes that:

“In fusing xeno approaches with speculative design, a practice of xenodesign can emerge, questioning who or what should be given agency in design, and for which reasons. It can become a practice of developing techniques for including multi-entity agency in design and for broadening both its human and non-human audiences.”  (Schmeer, 2019)

Struggle with implementation

To sketch our idea we are using a simple houseplant with an attached electrode sensor on it. Initially meant to be used as a wearable body tracker tool it becomes one for fauna. The electromagnetic current coming of a leaf is delivered to a computer via Bluetooth through from Bitalino hardware. It is then processed via processing software and with the help of Webinator we are able to connect sounds to the different states of plants electromagnetic currents. We have decided to train the machine based on three different samples using Wekinator software. Aiming for coherent sound output we wondered what is going to be the base, what is going to make up the rhythm and what sounds complement the presence of a plant. After exploring the changes the plant affords when touched we discovered that it has a stable electromagnetic current while touching its leaf allows for a slight increase in the input. Furthermore, if touched on the actual electrode, it produces the most intense input. It is nevertheless problematic because in our case, we are basically forcing meaning out of the noise. The tool used by us is meant for tracking a human heart pulse. Thus it can be seen as a kind of bad AI: a computer program that imposes meaning instead of transmitting the meaning already present in the data. 

[tangible-journal-blog]

Week 4, Tuesday

I am skeptical about the idea of interactive art performances that seem to center around simply mirroring the gestures/movements of a performer. I believe that such practices call of a deeper consideration of their purposes, especially when it comes to the use of technology implied. If we take the idea of a simple feedback loop between the body as input and a visual as output, in my opinion, it falls short in being a tool to think through. Therefore, I would like to use this week’s task to think of a way to go beyond direct control and towards a more nuanced, even poetic use of AI in interactive art. This interest of mine is largely inspired by the work of architecture and design theorist Benjamin H. Bratton, who, when discussing the shapes an ai can take suggest to consider it as "A kind of generative force at an ecological scale, one that augments existing intelligence and introduces new forms besides it.”. An example of artistic work which, in my opinion, embodies Bratton’s idea, is Gil Delindro’s  (UN) MEASUREMENTS - 2 # The weight of repetition. Here the artist uses an old tree trunk’s surface patterns as input for generating sound. Although not employing machine learning or ai more broadly, this piece of work is important to me because, unlike the idea of a feedback loop between human and machine, it brings the non-human, non-animal presence to the center of our attention. By explicating the patters of a tree trunk and making them heard through technology-enabled translation it shifts away from a human-centered panopticon to a meditative, even poetic plant creativity. AI tools are fruitful ways of creating complex interactions because they can be, as Bratton states, tools of augmentation in different fields of life. As presented by the course readings about the instruments, the instruments still function as analog instruments while presenting additional technological features (Macionis & Kapur, 2018).

Thinking back to the week’s readings, Bevilacqua, Boyer, Françoise, Houix, Susini, Roby-Brami and Hanneton suggest that “the design of the movement-sound can be leveraged by taking advantages of users expectancy on the auditory feedback” which the authors describe as an ecological relation between sound and action (2016, p. 4). But what if the user performing an action is not a human but an ensemble of human and somewhat unpredictable plant reactions?

When it comes to the plant agency to act, Elowan: A plant-robot hybrid the work by MIT media lab, illustrates how this abstract notion can be designed for (video above). What interests me in Elowan is the unexpected shift in agency which the flora gains when it moves and be present in a way animals and humans are: lively in movement. Another way things can be present is sound. And this is where our group’s idea comes in. For this week we choose to opt for designing a musical instrument out of a living houseplant. This is not a new idea neither is it a niche one: hardware for generating music from plants are both successful Kickstarter campaigns and contemporary sonic art pieces.

Bevilacqua, F., Boyer, E. O., Françoise, J., Houix, O., Susini, P., Roby-Brami, A., & Hanneton, S. (2016). Sensori-motor learning with movement sonification: perspectives from recent interdisciplinary studies. Frontiers in neuroscience, 10, 385.

Macionis, M. J., & Kapur, A. (2018). ’Sansa’a modified sansula for extended compositional techniques using machine learning. NIME.

[Strelka Institute/Институт Стрелка]. (2013, March 21). Strelka Talks. Benjamin Bratton "Alternative Models of AI (at Urban Scale)" [Video file]. Retrieved from https://www.youtube.com/watch?v=A3C31DhoPQ4

[tangible-journal-blog]

Week 3, Thursday

Concept

Inform and provoke by showing the imbalance between income levels and the property prices in Malmo. By engaging the user to act as a sort of “builder” of the city (making them take, feel the weight of and place the new property prices (cubes) and incomes (balls) for each year one after the other). Jansen, Dragicevic, Isenberg, Alexander, Karnik, Kildal, ... and Hornbæk (2015) write that  “Several TUI systems have been developed for such tasks, but they typically use physical objects as controls” (p. 4) while leaving out data to be presented digitally. They suggest that dataPhys, meanwhile, is focused on “information output and exploration tasks (with input being used to assist in the task)” (p. 4). Drawing upon their argument I suggest that our work this week focuses on exploration. It is nevertheless a controlled exploration because the person encountering our design presumably has to follow the dataPhys as if it was an open-ended board game: arranging ever newer blocks of data marking the past years' statistics. Ishii and Ullmer (1997) write that the coupling of bits and objects is a “Seamless coupling of everyday graspable objects (e.g., cards, books, models) with the digital information that pertains to them;” (p. 235).

Function

The function of our concept is to communicate the imbalance between income & property price levels, to sensitize people to the lack of citizen agency over this growth and to ambiently communicate the human flow in-out of Malmö.

Behavior

Jansen et al (2015) write that in “comparison to screen-based visualizations, physicalization can similarly offer to bring data to many more places and aspects of our social lives. In particular, since physicalizations can be anywhere and are always “on”” (p. 3). Following this notion, our design allows for the user to become a creator of dynamic dataPhys. The context of use for our design is an informal educational setting, where data is coupled with simple blocks via weight (making it a sort of an analog tangible-bit which is always “on”). As such, our design could be used to provoke an understanding and discussion about citizen agency over the rising inaccessibility to property in Malmo.

Structure

Our design consists of a board, shaped like the territory of Malmö and marking 3 districts (Centrum, Hyllie, Rosengård-Hussie). The board animates via flows of light, which ambiently display the flows (numbers) of people moving in and out of the city and inside it like currents in an ocean.

(source of the image the current pattern is taken from:  https://www.aoml.noaa.gov/phod/dhos/models.php)

Furthermore, weighted rectangles represent the prices of property. There are three colors that represent Centrum, Hyllie, and Rosengård-Hussie district prices. The weighted balls work in the same manner, representing the corresponding incomes. Both figures are meant to be marked by the year who’s data it represents.

Ishii, H., & Ullmer, B. (1997, March). Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (pp. 234-241). ACM.

Jansen, Y., Dragicevic, P., Isenberg, P., Alexander, J., Karnik, A., Kildal, J., ... & Hornbæk, K. (2015, April). Opportunities and challenges for data physicalization. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 3227-3236). ACM.

[tangible-journal-blog]

Week 3, Wednesday

When it comes to informing and provoking, the former is about providing information in an articulate and sensible manner, while the latter goes one step further and aims to evoke a reaction in the receiver. I believe this can be done both by choosing provoking (undiscussed, taboo, alarming, etc.) data and by opting for provoking (exaggerated, rousing, offensive) ways of communicating it. It must nevertheless be considered that provoking is a slippery way to take and can easily shift towards a flat stirring. Nevertheless, provoking as a thought-through strategy of engaging the receiver of information is more likely to include subtler nuances of ethical and moral implications. Hence if a designer does not consider their stance as political (which it always is, more or less), they fall short on reflecting on their own prejudices, biases, and privileges. Finally, if provoking is to be a subtle and considerate strategy of informing, it should also slowly give rise to certain affects (bodily and emotional) rather than rushing to provoke by a disturbing or flat narrative.

For this week’s assignment, we have decided to inform and to provoke based on Malmo demographics data. From this wide topic, we narrowed down to the idea of social geographies of Malmo, looking into data relating to how people move inside, from and to the city. Our starting point is looking and the movement of people. At first, the discussion centered around the force of property price as a crucial factor in influencing human flow inside Malmo and between its different districts. In such a way we could bring forth concern for gentrification and a lack of human agency in being moved/moving in the city. However, we have first decided to frame our work for week 3 as being about different groups of people moving in and out of Malmo (Moving in: from Sweden, immigrants / Leaving: free-willing, forced) (figure 1). Here our concept centered around provoking by making the user move tangible bits (Ishii & Ullmer, 1997) of data in and out of the city once they invite them to do so by either glowing or making a sound. Once joined to the city shapes pile (which appears in a children's wooden cube-like game shape, like in figure 2), they would produce a sound. In addition to that, we also had the idea of making the user move the shapes inside the city to represent the changing number of inhabitants in different areas. This would be based on the data shown in figure 3.

figure 1

figure 2 (source: https://www.michaels.com/0.75in-square-wood-blocks-by-artminds/10111041.html?pp=1&cm_mmc=zadv_PinterestBrand-_-pinterest-_-alpha_-2017&utm_campaign=alpha_626736582845&utm_medium=social&utm_source=pinterest&utm_content=2680060802882)

figure 3

However, at the end of the day, we reevaluated this idea. We are indeed imposing a certain privileged gaze and reestablishing the status quo of refugees versus non-asylum seekers if we follow the previously described framing without changing it. How might we provoke in a thought-through manner that does not impose a certain worldview yet highlights the facts of Malmo demographics? To be able to do so in less than a week’s time we asked: “How might we represent the rising inaccessibility to real estate in Malmö ?”. To do this we will aim to communicate the shrinking gap between income and real estate prices. As data demonstrates, although both numbers keep on growing, the rent price grows to make up more and more percentage of the income.

Ishii, H., & Ullmer, B. (1997, March). Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (pp. 234-241). ACM.

[tangible-journal-blog]

Week 2, Thursday

The puzzle of categorization

Visual perception is so instant and so embedded in our categorization process that we take our perception of differences and similarities of the original “Guess Who?” for granted. Nevertheless, when thought about thoroughly and in parallel to smell as a categorization tool, vision appears to be more alike to smell than one might initially consider it to be. For example, when one sees the “Guess Who?” board with over 20 faces with different visual characteristics for the first time a the start of the game, the categorization based on perceived gender, hair or skin color is very instant. This is because we are so used to distinguishing visual cues and ever more so human faces. In the original “Guess Who?” game it is typical to begin with the mentioned general characteristics. And although some characters are wearing glasses, hats or have bigger noses, for example, it is more rewarding to start with general questions (race/gender) because they allow to flip down a bigger group of characters instantly, making winning more likely. 

Drawing upon this, our group discussed how categories work when it comes to smell cards instead of human faces. Unlike when viewing, smelling does not allow to quickly notice and isolate groups of scents. Instead of this, a player has to prime themselves by inhaling each smell card, then memorizing and distinguishing possible categories based on comparisons and associations. Does it smell fruity? Does it smell like a spice? After trying the game out we have decided that categorization is initially hard, but as one continues playing more and more the smells become more familiar. Comparing vision with smell made me consider how vision is not as different from smell when it comes to processing input: a person has to consider all the factors (similarities, differences) first, then draw possible categories from these. An additional idea we had to help probe a player for categorizations was to use caps on top of cards. These would allow each player to make a visual mark (such as, for example, a colored dot) which could in turn help remember categories, speeding up the game and aiding learning. Caps also allow for a sort of overview-gaze over the board, much like in the original game, but based on visualizing one’s smell categories instead. 

The caps could be opted out or into using anytime, possibly not used anymore as a player develops a skill for scent. This reflects what Niedenthal (2012) describes as being “one of the essential tasks of designing smell into games”, namely helping “players develop a basic smell connoisseurship adequate to the challenges of the game” (p. 16), which also supports players in developing “specific cognitive skills, namely learning new mental categories, and how to fit new smells into them” (Gilbert as cited by Niedenthal, 2012, p.16 ). When testing our game we realized the caps are also useful in covering up the names of smells on each card, preventing a sort of cheating, since names can give up a category (for example, bloody orange and grape both being fruit). And since there is no cap on the card to be guessed by another player, seeing the title allows them to approve or deny the other’s guess too.

Throughout this week we have been struggling to conceptualize what the idea of encapsulation Niedenthal (2012) means in our design. Is our way of working encapsulation, if the word carries the term capsule in itself and our papers lack such a container layer, is this still it? On the other hand, if encapsulation is to be interpreted as simply meaning the capturing fleeting particles of smell in a stable tangible material, our work follows that. Here scents are encapsulated in simple water-color paper and delivered via vaporization/natural defusion while smell delivery is controlled by the players (smelling each card by approaching it with their nose).

Niedenthal, S. (2012). Skin games: Fragrant play, scented media and the stench of digital games. Eludamos: Journal for Computer Game Culture; 1, 6.

[tangible-journal-blog]

Week 2, Wednesday

How can we add a smell dimension to an existing game in a meaningful way?

Today’s workshop exploration of smell vortexes led me to think about how they are experienced in terms of meanings as associations to their way of delivering smell. I suggest they appear as intense, sudden announcement mediums. Much like a cannon bomb, they seem to be sudden, aggressive even, in their way of delivering smell. Smell vortexes also afford addressing each person directly by shooting the smell up to every one of them personally. On the other hand, these objects can also shoot smell out once and allow it to slowly spread in the surrounding space. Such thoughts arose in our group in relation to designing a Bingo game based on smells instead of numbers. The aspect to be replaced by scent would have been the announcements of number. However, after discussing the problematics of delivering smell directly and simultaneously to the bingo players (just like the sound of a number announcer’s voice is delivered) we changed our direction of work.

Our goal for this week’s task is adding a smell dimension to the 2-player board game “Guess Who?”. In this game, both players get a set of cards with faces of people and their names alongside one card with a person the other has to guess by asking yes or no questions to isolate a category of people based on the details of their looks. Each person gets one question per turn and an extra one after getting a positive reply to their question. We have decided to replace the faces of people (vision) with scents on cards (smell), meaning that categorization happens based on the characteristics of smells instead. In the original game, each player has one chance of guessing who is the person depicted on the other’s card by saying the person’s name. In our version of “Guess Who?” we opted to not have names of scents displayed and considered creating fake names or naming them with random alphabetical letters, leaving the original title invisible.

Guess Who? rules. Source: https://www.hasbro.com/common/instruct/GuessWho.PDF?fbclid=IwAR0XDM0vmeCk8h17g2NofOgvT0pJOoZAEnxde3TAFh7qEqWDx2mZSWTvquQ

[tangible-journal-blog]

Week 2, Tuesday

I believe that when it comes to interaction design practice working with smell provokes a re-thinking of the role and hierarchy of human senses embedded and reproduced through certain designs. For example, let us consider how ubiquitous touch screens have become and how that places many people in western societies on the verge of cognitive overload from visual input. Touchscreen surfaces are designed almost tactile-less and the ever-growing screen surface seems to be designed  exceptionally for the primacy of the image. Nevertheless, such practices, however dominating, open up design space for the overlooked senses, such as smell, to step up in the sensory hierarchy ladder and possibly replace vision. The central tension of such a premise presented by Olofsson, Niedenthal, Ehrndal, Zakrzewska, Wartel, and Larsson (2017) is whether smell technology can be purposefully/meaningfully incorporated into digital games and if so, how. Instead of making smell add to digital media, the authors ask “how can digital applications serve olfaction?” (p. 5). How can a simplified understanding of smell as adding to immersion-only be overcome? Niedenthal (2012) expands upon it: “A naïve understanding of immersion as sensory verisimilitude, however, offers one of the least promising avenues for the development of scent for games, because, as we have seen, it is difficult to synchronize scent effects to visual and aural media.” (p. 13). 

Niedenthal, S. (2012). Skin games: Fragrant play, scented media and the stench of digital games. Eludamos: Journal for Computer Game Culture; 1, 6.

Olofsson, J. K., Niedenthal, S., Ehrndal, M., Zakrzewska, M., Wartel, A., & Larsson, M. (2017). Beyond smell-o-vision: Possibilities for smell-based digital media. Simulation & Gaming, 48(4), 455-479.

[tangible-journal-blog]

Week 1, Friday

Final video 

[tangible-journal-blog]

Week 1, Thursday

storyboard

Is our work about behavioral feedback?

Gouveia et al. (2016), write that behavior feedback suggests a particular way of using bodily data tracking tools. They write that “first people collect data, then explore and review summaries of longer periods in retrospect (i.e., days, weeks) to identify patterns and plan alternative future courses of action” (p. 144). Putting our idea in comparison to this definition, while it is not about very strict self-monitoring for precise goals, as Gouveia et al. propose sports watches to be, it is nevertheless based on collecting data. Yet this is not about self-monitoring. I believe our group sees it as being about awareness via getting to know your bodily reactions. It is not direct behavior that is being monitored but the overall condition of the body as reacting to stimuli and situations (lack of sleep? food? too much coffee?). Furthermore, Gouveia et al. propose that supporting comparisons to targets and norms is another aspect of glanceable behavioral feedback: our prototype indicates a normal heartrate with a blue pattern. When it rises, it turns first light blue and then red, indicating that the rate is no longer “normal”. Finally, the authors propose that “effective glanceable feedback interfaces should not only inform but also instigate short, goal-related actions” (p. 147). In our case, it is about encouraging awareness that might lead to actions to relax, or maybe bigger lifestyle changes.

A reflection on the role of paper prototyping in design

After working with this method this week I believe that paper prototyping is perhaps meant for more experimental or conceptual design wireframing. Paper prototyping in interaction design is particularly enriching when it comes to representing movement in a low-fi yet divergent manner. For instance, in a scene at the end of our video prototype, a slightly broken-movement line of heart rate can be seen moving on a smartphone screen. Although accidentally not moving in a “smooth” way, it adds a vibrating effect to the feedback, similar to an experience of heart rate: something human-like, bodily, affecting rather than just illustrative.

Gouveia, R., Pereira, F., Karapanos, E., Munson, S. A., & Hassenzahl, M. (2016, September). Exploring the design space of glanceable feedback for physical activity trackers. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 144-155). ACM.

[tangible-journal-blog]

Week 1, Wednesday

Today we changed the scenario framing to be about a person monitoring their stress levels via heart rate instead of designing for a more “serious” health condition. The scenario as for now remains very similar to that described in yesterday’s entry: centering around providing notifications about a growing heart rate or a stable condition yet excluding the option of calling for medical help.

Our changed framing calls for a reconsideration of our prototype in terms of distraction aspect as described in the previous journal entry. Our smartwatch feedback is not designed to provide urgent information anymore. It is currently more about notifications on a state or and ongoing event rather than induced notifications popping up constantly.  

Abstraction of data

The problem with Gardy (Gouveia, Pereira, Karapanos, Munson, Hassenzahl, 2016, p. 146), in my opinion is that a garden is not an abstract representation of data in the same way as, for example. circles and other simple shapes might be. This observation influenced our group to abstract data via shapes and colors that shift according to changes in heart rate data. Our choice of abstractability ties into the aspect of user satisfaction: by providing an option to switch between different modes of visually abstracting data we attempt to allow for agency over the interface mode in a simplified way. The gif below illustrates how we are proceeding with this in our work.

Gouveia et al. group Gardy together with more literally abstract representations, but I suggest that it needs to be distinguished as a metaphorical depiction instead. This is because, first of all, a garden blooming, unlike simple shapes, consists of quite many forms which visually that represent it. It is interesting that there seems to only be an option for growing a garden and not destroying it. Perhaps expanding upon the narrative would allow for more user satisfaction when it comes to interfaces similar to Gardy.

[tangible-journal-blog]

Week 1, Tuesday

Our framing for this week’s task (as agreed on today): a person monitoring their heart rate for health reasons with a smartwatch and a smartphone. The scenarios of use we initially wanted our paper prototype to illustrate were interacting with an increased heart rate warning, high heart rate warning as well as regular monitoring of one’s condition. A loose outline of how these 3 use cases intertwine is shown below. The principle of our idea is to provide a notification on a watch and to follow with more information on a smartphone screen, allowing to reflect on data in comparison to that from the past (last week’s or yesterday, for example). We are also considering to make the interaction more engaging and unique by allowing for simultaneous viewing of the same data abstracted to a different level on both screens.

I will now continue with evaluating our framing and budding ideas against the key characteristics of glanceability, as outlined by Matthews, Hsieh, and Mankoff (2009).

Learnability / By simultaneously looking at a phone and a smartwatch screen, both of which demonstrate less and more abstracted data. Respectfully, the user can learn to recognize how simpler patterns reflect specific numbers or statistics, which builds awareness (awareness being ”the amount of information shown by a display that people are able to register and use.” (Matthews, Hsieh & Mankoff, 2009, p. 453)). Here the notion of learnability ties into what Mankoff et al. refer to an as easy transition to more in-depth information (as cited in Gouveia, et al., 2016, p. 455). In our prototype, one can proceed from notification to more information in one click, which reveals, for example, a number indicating what the abstract figure on a watch screen refers to data-wise. If clicked again, data in more depth appears on a smartphone screen. This also ties into user satisfaction if viewed in terms of our prototype allowing for customization.

Error visibility and recovery / Our interface allows for errors to be visible since it is always in motion, representing an ongoing process of a heartbeat. According to Mankoff et al., “error prevention” should be considered in design because “users should be able to distinguish between an inactive display and a broken display” (as cited in Matthews, Hsieh & Mankoff, p. 452). It follows that if the screen froze, that would likely inform a user about a functioning issue, distinguishing it from a functioning interface ( Matthews, 2006). I am still unsure about how to work for ensuring an easier recovery from an interaction breakdown from the user’s side. What actions might be taken to restore the monitoring? 

Usefulness /  Matthews, Hsieh, and Mankoff (2009) suggest that usefulness is the level to which a display enriches the experience value to the user. Authors elaborate that peripheral displays should communicate information in a fine manner while and avoid distracting the user from what is going on. They point to an important tension that exists when designing for the usefulness of a display: namely that too subtle of a display can prevent an interface from communicating the urgent data (p. 452). Thinking back to this puzzle, I propose that while our prototype does communicate through abstraction, it is nevertheless focused on the most urgent data: the current, live reading of a person’s heart rate. It is not a comparison with an average heart rate (which is presented in the phone screen as an additional nuance) that is the primary information to convey. 

Distraction / Our smartwatch feedback is designed to provide urgent information on behavioral/health state and to also remind the person to reflect on patterns in their health. It can thus be seen as a positive distraction encouraging behavioral changed. 

Gouveia, R., Pereira, F., Karapanos, E., Munson, S. A., & Hassenzahl, M. (2016, September). Exploring the design space of glanceable feedback for physical activity trackers. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 144-155). ACM. 

Matthews, T., Hsieh, G., & Mankoff, J. (2009). Evaluating peripheral displays. In Awareness systems (pp. 447-472). Springer, London. 

Matthews, T. (2006, June). Designing and evaluating glanceable peripheral displays. In Proceedings of the 6th conference on Designing Interactive systems (pp. 343-345). ACM.