me-more-y

[me-more-y]

WORK IN PROGRESS

This website will be updated on an ongoing basis and updated on future releases.

[me-more-y]

Introduction

We are our memories. And our feelings, and our plans for the future. Those who are not here anymore, only exist in our memories, and in our hearts. The past only exists through documentation, imagination, recreation, evocation, and recollection of impressions. History is the sum of individual and collective memories and their commemoration.

Me|more|y is an interactive installation made of several pieces. Digital and physical symbolic representations of unrelated memories without any proposed narrative.

Me|more|y is an autobiographical piece and an exploration of the vast field of memory. Based on personal experiences of close relatives with dementia. On the personal difficulty to remember people, events, and places. A continuous state of frustration to retrieve information. Me|more|y  is a reflection and representation of memories of an entire life.

How differently do we encode, store and retrieve memories in the digital era? How differently our brains operate compared to machines and computers? Our memories are fluid, translucent reminiscences. Sometimes sharp and clear, but most of the time hidden or covered by a blurry mist.

This project is dedicated to my grandparents. To Maruja, who lost her memory and lived in a limbo state for almost ten years. As his father before, as her sisters later on. To Dionisio, who accompanied her in all those years until he decided to end their travel at the sunset of their lives.

[me-more-y]

Concept and Research

In this project, art works as a cathartic element that helps cope and deal with extreme emotions. As a way to deal with vital matters-concerns that provide relief through the representation of these emotions. Opposite to Design, used to solve problems, Art is all about questions, to propose those and the look for the answers in a very personal way.

Me|more|y benefits from the Design Thinking and Human Centred methodologies used in the creation of digital products and services. Different activities were held in through the creation process. These are the phases: Discovery (Define the challenge, research, observe people, Insights...), Define-Describe (Hypothesize), Design, and Iterative Delivery (Prototype, deliver, re-describe, re-discover, test and learn).

The project started with the Discovery phase, diving and absorbing all the information related to the memory in a very broad sense. Reading materials available from very different fields: science, literature, novels, mnemonic techniques, movies, art specific material through the time… Researching on social media as on how people perceive and visualise memories. The field of memory is huge, very broad in the different topics.

In the art field of memory, these are some of the artists and pieces that influenced my work. From the autobiographical documentation work of life and memories by Frida Kahlo’s self-portraits to Tracey Emin autobiographically based works, or Nan Goldin albums to Christian Boltanski recreation of IIWW memories  (“Contemporary Art and Memory - Images of Recollection and Remembrance by Joan Gibbons) to specific pieces as:

Cesar Kuriyama documenting his life and creating a video with a recording of a second of every single day in his life. 

Chloe Meineck with her music box to help dementia sufferers to retrieve memories through music. 

David Szauder, creating glitch images to represent fragmented memory and the imperfect nature of human memory. 

LaMen. The image Memorability project. The largest annotated image memorability dataset. Using Convolutional Neural Networks (CNNs), to create memorability maps of images. 

The work commissioned by Swarovski to Troika: Hardcoded memory: “Digital crystal: memory in the digital age” 

The full list of artist can be found on the References section.

From the beginning there was a clear strategy, the decision was to develop different tasks to exercise the memory,  becoming the process as important, if not more, as the outcome. Every task would help to retrieve and analyse different aspects of memory.

The work echoes insights from one hundred and seven responses to a proposed survey during the research phase. The questionnaire was answered by 46 in Spanish and 61 in English all from very different nationalities and backgrounds. Some of them I met before, some were friends, other relatives, friends of friends I’ve never meet… The power of social media. The survey touched different aspects for the memory: earliest memories, specific memories in time, random memories, possibility to chose memories to forget in case of dementia…

Following the research, the next step was to create individual prototypes that would touch a specific area.

Me|more|y is a representation of different aspects of the memory. These are broad, complex, sometimes overwhelmed. The project is divided into three main parts, similar to theatre acts. In this case, no specific order was required. The particular sections were defined at the beginning of the project and adapted lately to the gallery space. The sections were: an interactive piece with different experiments, a physical representation (transparent printouts) and the infographic mapping all the people meet in a life’s time.

The project is purely visual, as my personal memories are. In the process of designing the outcomes, I’ve made the conscious decision to avoid other senses as the hearing, smell, touch… The reason why is that I cannot recall any olfactory, tactile memories...

Physical space: The room was set up to create an immersive experience. Each wall holds a different outcome. In the center of the room, a plinth. Where the spectator interacted with the main section. The place was divided into two areas with two different wall colours. They represented the lights and shadows where we move when we try to remember. Two walls were painted in dark grey color (Deep Space Echo http://m.valsparpaint.com/color-detail.php?id=103537&g=100037). This colour enhanced the graphics on the screens. The other two walls were painted in white. One of those was the receptacle of a marvelous window gallery. The room was not illuminated. During the day the natural light passed through the transparent printouts evoking religious echoes as a spectator defined in his visit. When the sun was set, the only light in the room was the one from the screens and a small light on the floor is hidden on the window that created a dramatic light through the transparent acrylics.

Interactive Piece - P1

The first piece (from now on P1) is an interactive screen-wall consistent of six 22” monitors. It displays four different representations of aspects of the memory. These themes are:

Retrieve memories. When trying to remember something the memoirs appear blurry, cloudy fuzzy… This exercise represents how our brain tries to retrieve these images. They seem close, they merge, they connect, once they seem connected, they become undone, dematerialized for a moment to become an image again. In the original survey one of the questions was: in the hypothetical case you would lose your memory, could you list the things you would like to remember? Each of these six images represents what I want to remember: my partner, my family, my travels, family/friends together around a nice meal, the place where I came from, and my favourite hobby (reading).

Unveil memories. Our brain stores all our memories, all the possible combinations, everything is there waiting to be retrieved. Sometimes we have flashbacks, and memories appear randomly without ourselves recalling them, sometimes in a more structured way. These animations represent all the possible memories that exist stored in the brain. But they are blurry, hidden, waiting for the right moment to appear. Is there, where images became sharp and they get the meaning.  In the first part of the animation, this appearance is structured. In the second part, the retrieval happens randomly. Sometimes we cannot control what it comes to our mind, in uncontrolled flashbacks. 

Digital Memories. In the creation of the different prototypes, I used all the digital photographies I have. In total 16.758 digital photographies. Since the very first digital image, I took to July 2017. My family scanned also photographies taken where I was a child. The arrival of digital photography works as a support of our memory, as an extended mind. Everything is recorded, and experts study how this affects the neuroplasticity of our brains. Different composition with all these digital files was created, following machine learning categorization methods. Each of the six images shows a different area of one of the compositions. One that categorize ALL the images: Food - People - Landscape - Nature/Flowers - Buildings - Night.

Non-digital memories.  What happens with the non-digital recorded memories? How can we retrieve them? Memories are recreation, the imagination of that what really happened in the original moment. These six animations represent six memories only existing on my brain (my first memory, the only tactile one, the awareness of the possible loss of a beloved one, the first time my grandmother did not recognise me…)  and how they fade until they disappear. Creating in the process new words that have not existed previously.

Collective memories. One of the questions of the survey was to name the first thing that came to one's mind related to a theme (a place, a meal, a historic event, a book, a movie and a song). After that, the request was to take some time to reflect on the favourite place, meal etc. The exercise consisted in identifying how the first impromptu thought differs to the ones that require more time. Some of the answers were a surprise: a big percentage of people from different backgrounds and nationalities have the same answers related to food. Same books in different languages appear as an answer, probably influenced by school literature and popular culture...

Ergo Sum. Infographic - P2

The idea for the second piece (from now P2 and titled Ergo Sum), it started a couple of years ago. It is an allegorical representation of all the people I met in my life. A video shows a 2D view of the infographic with all the names visible at the same time and a 3D render of the same datasheet displaying a 3D shape for each of the people I met.. The look and feel are aquatic, reminiscence of the deep waters where I come from.

The process reveals itself to be just as important as the outcome of the whole work. The datasheet to create the list was done manually by choice and it is chronologically created. Family, primary school, secondary, university, different job roles… There were cases in which I exhausted the names for a period, so I contacted someone to help me to remember. That created a chain of connections and refresh of old relationships. Everyone was keen to help. In other cases, I used the help of technology. For instance, I could export the full list of my LinkedIn connections (1020 people at this moment) in a couple of seconds. But instead, I copied manually one by one every single name to help me to exercise my memory, because that was the project is about. This process took several days instead of seconds. Again I am reflecting about how the brain works compared to a machine.

Physical Representation - P3

The third component (from now P3)  is the physical representation of memories: translucid, complex. From the very beginning, I decided that the final outcome will be displayed in a transparent material and would be hanged from the ceiling. Through the months, different experiments were created to test different memory concepts and technologies. Finally, four were the representations chosen to be printed. All of those created specifically for that medium and not represented in the computational animations. A 76x76 segment of a representation of all the images stored digitally in all my devices in a grid mode. A 50x50 image of images in a cloud model, a 50x50 and negatives and positives of eye detection algorithm... It is really interesting to see how the machine needs different trials and adjustments to get accurate results and has a “similar” behaviour as our brain. The human needs to supervise the computer’s outcome for an optimal result. In this specific piece, both positive and negative results for eye tracking are displayed. The last small piece is an early test from the P2 Unveil memories.

[me-more-y]

Technical documentation

Me|more|y uses a variety of technologies and languages. I learned some of them from scratch, starting with online learning courses (For instance, Udemy for Python) and then following an innumerable number of tutorials online. The main languages and frameworks are Processing, Python and d3.js. With several techniques used such as face detection, clustering images with t-SNE... All the physical computing is handled by Raspberry Pi computers. Please refer to the technical References section for further detail on specifics.

P1 - Interactive

P1 consists in an interactive Pi Wall of six monitors connected together. The interaction with the Pi Wall is done with an iPad through OSC communication. Each of the six monitors is connected to a Raspberry Pi 3. Once each of the Pis has been installed and set up (Noobs, ….) Inside each Raspberry Pi the following code has been included: OMX Player and pyOSC

The installation displays four different animations. Each monitor has a different version of one of these four animations. That makes a total of 24 different animations.

Once the viewer activates each of the animation clicking on an IPad, the iPad sends a message to the first Raspberry Pi. Pi01 behaves as a master and send the activation message to the other five via OSC. I’ve added a button with a reboot code, to press before shutting down the installation, then the only action needed would be to unplug it from the power. In order to switch on again the piece, the only actions required was to plug the power switch on.

In order to ease the performance and try to avoid any possible real-time display issues, the computation happens beforehand in a laptop. Then is recorded as a video (this is what the piWall displays). In the early prototypes, I installed processing to run the animations directly on the Pi, but the processor was not powerful enough. I tried also with Openframeworks, but some of the animations have to process more than 16.000 images and this can take quite a time to render and process.

In order to create the four different animations, different techniques were used.

Retrieve memories. It uses Processing and  Shaders to represent the movement. Each photography is divided by colour channel and a white and black filter is applied.

Unveil memories. For the creation of the background in this image, I used Openframeworks and tSNE. The animation itself with the blur and the unveil parts is done in Processing, using blend modes and Arrays.

Non-digital memories.  The animation is created in Processing. Each memory is coded as a separated String. At a specified frame rate a character is subtracted randomly creating new words. This happens each n number of frames until the text disappears completely. To display the background this piece uses a noise animation. The frame with the photography is created with masks and filters. Each one of this frames use a photography taken by myself of a different sea at different occasion (storm, peaceful…)

Collective memories. As the previous bit, this animation was created in Processing. It uses mask, filter and blend modes to get the different effects. These displayed here are the only images not taken by myself. I used Python to scrap all the images from Google, being a big percentage of free usage rights. I scrapped in an earlier phase of the project using Python as well images from open accounts on Instagram, but these were not relevant for the purpose of the piece. In order to access to each of the single Pis in an optimal way, I used iTerm to access and control them in a single view through my laptop.

P2 - Ergo Sum

The video recorded uses two different infographic representations created using d3.js, a data visualization library. The first one is a 2D representation and the second one a 3D version. To render the 3D view, Three.js has been used on top of d3.js. In both cases, the infographics are rendered by loading a JSON file that includes all names. In the different prototypes, the JSON was stored locally. But I connected the code to an online version stored in Google Drive, so each time the spreadsheet was updated, this will appear in the rendering.

P3 - Physical representation

I am very interested in learning how Machine learning and different AI algorithms works. For these representations, I studied Gene Kogan’s Machine Learning for Artist. ML4A. His work and publications have been an invaluable source of knowledge and inspiration. I have tried all his different tSNE prototypes as Clustering images with t-SNE, Image t-SNE viewer, Image t-SNE live... with different configurations and features (Mosaics, Grids…).

The grid and cloud representation printouts displayed the categorization of all the digital images I stored in my life (more than 16.000 images). For that, I used Openframeworks and t-SNE.

One of the key challenges of these pieces (the same as in the Collective Memories and Unveil Memories above from P1) was to scale the prototype to use the final number of files (more than 16.000). I collected all the images that came from very different sources, sizes and resolutions and unified them. This was a big task in itself. Keeping them with the same size, resolution and even name convention.

For the printout that represents all the eyes in a grid, I started extracting all the faces from the images in my library and then moved to extract the eyes. I used python and Face Detection using Haar Cascades - Face Detection OPenCV. I also included on the bottom of this printout, the false positives images the algorithm recognised as eyes. I supervised manually the outcome to improve the computational work done by the machine. As the brain, the machine also makes mistakes, and as our brain, similar memories are visualized as similar, even if they are completely different.

[me-more-y]

Self-evaluation

First of all I would like thank to all the people who took the time to answer the questionnaire. The responses to the survey were very varied and surprising. People contacted back saying they have never reflected on such deep questions. Some people said they were very touched reflecting such things… Other opened their hearts and soul to confess very personal statements.  I was moved and shocked reading such a personal confessions and how people opened their hearts and souls. At sometimes almost crying reading such sad statements… others laughing…

Someone said that being in the exhibition room was like being in the brain of a person, complex, overwhelming… This was one of the main goals of the piece.

Growth and learning most of the times came through pain and facing challenges and being outside the own personal comfort zone. Sometimes painful, we should embrace these challenges. These will be the ones that will make the next piece better. I faced several ones in the creation of this piece. Some of them very unexpected. The most difficult ones were the ones outside my scope and control, based on other people dependencies.

The first and obvious one, the conciliation of work with the creation of the project. Not having material time to access to the lab influenced the decisions to create the outcomes and the elaboration of those. For instance, I’ve created a wood and very simple structure to frame the six monitors. With more time and budget I would create a metal structure.

Dependencies. One of the unexpected ones was the access to the monitors. By early conversations, I was hoping to get them through the Department. I love to plan in advance and not having access to these one month in advance to the exhibition stressed me a lot. I decided to change my strategy and get them through other channels. I tried different things, everyone of those a different adventure and I finally travel outside London to buy second-hand ones.

Collaboration. Not being independent to do everything by myself was one of my big frustrations. Even with a ladder, I could not drill the ceiling! I could not lift myself the monitors when assembled together (6kg each x 6 monitors one plus circa 5 kg of cables). I was lucky to have the help of colleagues, Pete (our incredible technician) and family realising these tasks.

Space. Having limited access to the gallery space (mezzanine only available from 10-5pm and not during the weekends) was one of the biggest challenges in order to test the piece and prepare the room. The room itself was in a very poor state, painted in pink and with innumerable holes in the wall. It took me a full day to restore the window. Half day to cover the holes. One day to paint it. And this only to be able to start to set up my piece. I tried to keep the screens and material at home until the very last minute where I was testing the piece at nights.

Test in the final space. The final piece assembly was not ready until very late. When I arrived in the church I faced several issues connecting the microprocessors to the wifi. The DNS and servers were bouncing and the code did not work. I finally overcome the issues creating a local network using my home router.

Scale. The first prototypes were created with a small number of sample images, in an incremental way. First 200, then 600, then 1000... But running the code and processing more than 16.000 images was quite of a challenge. tSNE with Openframeworks and that number of files was very slow, up to a couple of hours to render. Python appeared as a faster and better solution in this case.

What I would do differently? A couple of people told me to create labels to explain each single piece. They were amazed when I explained each of the pieces to them. Without me explaining it, some of the magic was lost. I thought a lot about this. At the moment I am still not 100% sure of what I will do in this sense. At the end of the day, I created this piece for myself, to feel alive and I am very keen that could interpret in their own way.

Committed to the original concept and outcome was another of the challenges but at the same time one of my principal goals. I pushed myself to be very strict in this sense. I believe this is one of the big successes of this project. It is difficult to sell an idea without having a tangible prototype but being confident to deliver what I proposed it was key.

These above are great learning lessons that I will bring with me to the creation and exhibition of future pieces. At the end, the biggest achievement and gratification was the feedback from the viewers and the number of questions I received. People were very interested.

[me-more-y]

Technical References

General - Network

Setting WiFi up via the command line https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md

Static IP Address https://www.modmypi.com/blog/how-to-give-your-raspberry-pi-a-static-ip-address-update

Monitor Specs https://www.cnet.com/products/dell-ultrasharp-2209wa-lcd-monitor-22-series/specs/

P1 - Interactive Piece

Video wall

OMX Player - Install Wrapper

Raspberry Pi

How to shut down or restart your Raspberry Pi properly 

Installing Pi 

How to display your Raspberry Pi’s desktop on a Mac 

Format Pi - SD formater 

Media Player

Split GPU memory 

Install Openframeworks in Raspberry Pi

Include startup button

Raspberry Pi: Launch Python Script on Startup 

OSC Pi5

Download library OSC pi5 

OSC and Network Communication, Processing - Postage Stamp Networked 

Touch OSC Mac editor 

Raspberry Pi and OSC: Part 1 

pyOSC

Python

Video Wrapper for using it in Python

Server on computer to view on local - python -m SimpleHTTPServer 

Images

Batch Resize Images Quickly in the OS X Terminal 

Instagram scrapper 

Tidy by size images in folder 

Mac OS X Finder filter image by orientation 

Changing file extensions for all files in a directory OSX

Processing Shaders

P2 - Infographic Ergo Sum

D3

D3.js  is a JavaScript library for manipulating documents based on data.

A Simple d3 Network Graph

Force Layout with Mouseover Labels

Force-Directed Graph

Multi-dimensional D3 force simulation

D3 Tutorial

Include texts in d3

SVG Window resize

JSON

SheetAsJSON - Google Spreadsheet Data as JSON

Exporting a Google Spreadsheet as JSON https://gist.github.com/pamelafox/1878143

Export gs (google script) into Python

Three.js

HTML - CSS

Mix-blend-mode

Basic CSS Blend Modes

Background size

P3 - Transparent Printouts

Gene Kogan’s work and guides. Github

Mosaic Used in early experiments not displayed in the exhibition 

Openframeworks

ofxCcv 

ofxTSNE

Python

Install Pillow library - pip install pillow 

numPy (vector) multidimensional arrays to save images (tsne) 

Face Detection using Haar Cascades - Face Detection OPenCV 

Object Detection: Face Detection using Haar Cascade Classfiers     

Face Detection on Your Photo Collection in Python 

AdaBoost, short for "Adaptive Boosting" 

[me-more-y]

Future Development

Me|more|y will be the exhibit in October as part of the collective exhibition Mortality-Virtuality in the Ugly Duck. Based on the feedback from viewers and, in order to adapt it to the new space, I will modify the outcome. One of the things I will probably do is to project on a wall in a big scale instead of display in a 50” monitor the piece of Ergo Sum. Most of the viewers were very surprised by this piece, some of them were very interested in interacting with it to have the opportunity to navigate it. Collecting this feedback with my original idea of giving the viewer the possibility to give the opportunity to add them to the datasheet, in the future I will work to create an online version where users will spend time on the piece. A couple of people were also interested in the idea to create for them a representation of their own memories as the ones hanging from the ceiling.

I am in conversations at the moment to bring this project to display it in Spain, where again will modify the piece, from the computational side and the physical representation to evolve it and adapt to the new reality.

As an artist and human being, my preference is not to spend too much time with the same specific theme. So I will move on in the search of a new one. The main macro theme of my work is myself and my view of the world as I commented earlier to deal with my existential concern and heal my soul injuries. I decide the pieces based on impulses, guts, and new tools-techniques-technologies I want to learn. These are my motivations. For me, the computational aspect is a tool. The code is the material I use to create experiences to express my questions and answers.

Vague images appear in my mind, I think them over through months and years… and at some point, I will work on it, as it happened with Me|more|y. What is pulsing lately: the loss of a beloved one, the cycle of life, the view of very black lava stone and a very fluor red-orange colour. So bright that will hit the viewer.

[me-more-y]

References

Art and Memory

Kader Attia reflects on memory with mirrors + more at Galleria Continua 

Memory with perfume by Igangme from Korea 

Troika: Hardcoded memory

The elderly share their most vivid memories; Jung recreates the scene. 

Hungarian digital artist David Szauder (known as Pixel Noizz) explores the imperfect nature of human memory by glitching old photographs 

How Sci-Fi invented the Superman Memory Crystal

Layered Portraits capture Memories of Famous Faces 

How memorable is your image? Now There's an App for That 

Memory Art in the Twenty-First Century 

Books

Memory: A Very Short Introduction (Very Short Introductions) by Jonathan K. Foster  

Form+Code in Design, Art, and Architecture by Casey Reas | Chandler McWilliams | LUST 

Contemporary Art and Memory: Images of Recollection and Remembrance by Joan Gibbons 

Memory - Documents of Contemporary Art: Memory Edited by Ian Farr

The Man Who Mistook His Wife for a Hat by Oliver Sacks 

Awakenings - by Oliver Sacks

The Pillow Book of Sei Shonagon by Sei Shonagon 

Documents of Contemporary Art: Systems Edited by Edward A. Shanken 

The Art Of Memory by Frances A Yates

The Curator's Handbook: Museums, Commercial Galleries, Independent Spaces by Adrian George 

Articles online

Your brain stores imagination and memory separately

Snapchat never forgets with new Memories feature

Alzheimer's disease and dementia: what's the difference?

Scientists use AI to 'rewrite' painful memories in people's brains

How to hack your memory and remember almost anything

One second every day

Music Memory Box helps dementia sufferers recall their past

Design and the Elastic Mind 

The Role of Consciousness in Memory

Watch your raw memories become mind-blowing abstract art

MIT News - Memory

Scientists identify neurons devoted to social memory 

How the brain builds panoramic memory

Photographs and Memories 

How to become a World Memory Champion in ten steps

Tangible Memories 

The IDA Model of cognition

AI Revolution

Artist's Memory Loss fuels discoveries about the brain 

Exploring the Link between Art and Memory

You can probably use deep learning even if your data isn't that big

(MLPs) - MLP has about 120,000120,000 parameters multilayer perceptrons

(CNNs) CNN has nearly 200,000! Convolutional neural networks

Synthesizing the preferred inputs for neurons in neural networks via deep generator networks        

Design and Look and feel

Typography: Typo Pro Amble and Cooper Hewitt 

Exoplanets

Illuminated code from Space 

Hand-Cut Collages Collapse Modern Disarray in North Dakota | #50StatesofArt 

AI Experiments: Translator. Neural Networks 

Data Visualisation

Sound

Granulated Barrier Reef 

Deep Sea Ambience

Underwater sounds loopable

Logistics

How to make a TV Wall Mount

I would like to show my gratitude to my colleagues Howard Melnyczuk, Amy, Cartwright, Arturas Bondarciukas for the documentation work they did and shared. To Juan Real for the sound and music and all the support during this year. To all the people who filled the survey that filled this project with content. To the Goldsmiths lectures and technicians that helped through the process and realisation.