preservedlemons-blog - Tumblr blog

[preservedlemons-blog]

Detecting vs Tracking

Its one thing to detect things using artificial intelligence, however tracking them is a totally different story.

Detecting and object happens in every frame of the video, this could be 20 times per second... or 60 times per second. What i had originally envisioned was every time a vehicle passes through a central region of the screen, it adds one to a variable. This variable would then be used to put data into a database. However, i quickly realized i was quite naive in thinking it would be this simple.

The whole program is controlled by a while loop. This loop is running very fast indeed, so if a vehicle is traveling slowly, and passes through the central region... it can add to the variable numerous times, creating inaccurate data. Inversely, if a vehicle is traveling too fast, it does not count the vehicle.

A failed solution i tried was to create a central region in the frame, to increase the area in which vehicles are counted... however this was a double edged sword, as it also increased the chance of vehicles being counted too many times.

This is where i realized the difference between detecting objects, and tracking them.

In order to track and count an object accurately, you first need to work out the central point of the detection (region of interest). This is simple enough. Next you need to assign a unique ID to that region of interest (ROI). You then need to work out in which direction the center point of that ROI is traveling, this is fairly hard... you then need to calculate the euclidean distance between the center points of that same ROI, as it goes from one frame to another. If the distance is short enough, assign the same object ID to the region of interest in the next frame. This also allows you to work out in which direction that ROI, and thus, the object itself is traveling (in my case, this is left or right (east and westbound on the motorway)). Knowing in which direction the object is travelling, and knowing which side of the frame the object is on, you can tell if it has passed through the center of the frame. This then allows you to count that object accurately.

Logically, not too bad, but in practice, very difficult. This is where i have enlisted the help of pyimagesearch.com .

I discovered a class which basically does exactly that. A link to the tutorial i have followed can be found here: https://www.pyimagesearch.com/2018/08/13/opencv-people-counter/

As of last week, i have semi-successfully integrated the class in the code above, into my program. However i am still encountering problems, mainly in calculating in which direction the vehicles are moving. Today i will begin re-organizing my code which has to date been separated into functions and classes.

Below is the display function which is where the magic happens, and i believe this is whats also causing the issue:

I believe the problem to lie in the following area: 

You can see the print statements, and what they should be printing, however the final one... “print(y)”, does not print anything. You can see that in the below screenshot:

This means the program is not getting to the “else” part of the if statement. And this is where the problem lies. I believe because this is a function, which is being executed inside a loop, with all the variables defined inside the function, it is running through the code and never changing the value of the centroid parameter of the trackableObject (i have definitely worded this awfully).

I will restructure the code, and report how it goes.

[preservedlemons-blog]

The model works!!

After successfully training the model, and testing it... it works, and works surprisingly well which is tres bien! (it also detects motor bikes, however these are rare on the motor way and could not capture a screen shot with all 4 classes in)

[preservedlemons-blog]

Let the training begin...

On Friday i started training. I first attempted using the original Yolo configuration file, which as it turns out, is extremely heavy on the GPU. Every time i ran it, my GPU ran out of memory, and the program exited. Because of this, i changed to the “Tiny Yolo” configuration. This did not overload my GPU. I set the number of epochs to 500, and let it go. Alarmingly, it was taking +5 mins per epoch!! This resulted in a total training time of 35 hours and 11 mins.

[preservedlemons-blog]

What data? Complexity ++

This week i’ve been thinking a lot about the different types of data i could generate from my system. The data could be in the form of wait time (how long does a car stay in the frame for?), unique cars to pass through the frame, total vehicles (per hour, half hour, minute... etc), and how many of each vehicle (car, van, HGV, motor bike), when is the road busy, which lanes are most used, etc. 

With each genre of data, there are also many different possible ways of actually displaying this data. 

I definitely want to count a “total vehicles”, as that was the main focus of this project. 

Recently, i have noticed a lot of traffic building up on the slip road, this is possibly due to a 3 or 4 way control further down the road. From this, i like the idea of “wait time”. To measure wait time, i could measure how long it takes a vehicle to travel the distance of the frame. To achieve this, i would obviously need to measure the frame (i’d have to go down to the road and measure it physically). I could do this based on how many frames a vehicle is in the Field of View (FOV) of my camera for. This means i would have to set the frame rate of my camera, however doing this could be dangerous as it could cause me to loose data (if i set it too low). To counter this, i would have to work out the Nyquist frequency so as not to cause aliasing. To do this i will have to come up with a “max speed”... maybe 80 mph, which is 35.7632 meters per second (i will measure the width of the frame in meters).

I will also need to find a way of timing each instance of each vehicle, and then adding that information to the database. 

[preservedlemons-blog]

Custom data set

To increase accuracy, i have created my own data set. My logic is that if i train it only on images taken from the webcam, looking out of the window, it should be much more accurate than using a pre-trained model. 

I have written a program which uses Open CV to take pictures every 5 seconds. So far, via this method, i have created a set of 1.7k photos, of which i have labelled 1.1k. To label each image, i have been using a program called “LabelImg”: https://github.com/tzutalin/labelImg

The program is very effective, allowing you to highlight each instance of each class, and then save the labels as an XML file. 

Be advised: the above XML file, and image are not related, however it shows the process. You can see the name of each class, and the coordinates of each bounding box. The XML file and image are both fed into the neural network (NN). The XML file tells the NN were to look in the image to find that class. 

There are only a couple of issues which i have envisioned. These are: night time, and the weather. 

At night, obviously light levels are much lower, and thus i have to up the brightness settings of the camera to counter the low light levels. As for the weather, a problem will be rain. The rain on the window will 1. cause the camera to refocus. 2. it will get in the way of the cars, and could cause the NN not to pick them up. For the auto focus, there may be a get around in Open CV, however the rain getting in the way of the cars could be a major issue. You can get “rain repellant” for windows, which may have to be an avenue i pursue. 

[preservedlemons-blog]

Yolo Object Detection

After an extremely long time, i have successfully compiled and ran the Yolo object detector. The example in the video below was using a pre-trained model, which came with the detector. For this reason, its labelling “vans” as “cars”, every now and then it’s detecting an aeroplane, cellphone, etc. I need it to be much more accurate than this. 

https://youtu.be/25EiGLddI1A

[preservedlemons-blog]

Data Flow - Visualising Information in Graphic Design

Talkshow 03-09-07 / Rhythm Textures Poster: 

On the left - Talkshow. Created by Catalogtree in collaboration with Lutz Issler, this piece aims to transform social and infrastructural data into artwork. It shows the mapping of a talkshow between Aaron Kobline, Wilfred Houjebek and Catalogtree on 03-09-2007

On the right - Rhythm Textures Poster. Here, Stefanie Posavec analyses the rhythm and flow of text, by representing every exclamation and pause in the thickness of the lines, or in their absence all together... just as the rings inside a tree tell us information about the seasons. 

I really like the idea of visually representing something that is as “un-organic” as a motorways traffic flow, in a form which is as “organic” as the rings found inside a tree.

In my case, the data could be shown in a similar grid formation, where the 24 hours of the day represents the X axis, and (possibly) the 31 days of the month representing the Y axis, and then representing the number of vehicles (maybe) in rings in multiples of 10, 50 or 100 (depending how large the recorded numbers are).

Research in the Visualisation of Financial Data: 

Created by Cybu Richli, this piece assess the risk of investment portfolios from different asset classes. The traditional 5 components of an investment portfolio make up the 5 axes (bonds, cash, hedge, real estate, stocks). 

I like this way of representing data by creating “blobs”, and then overlaying them in 24 different hues which represent the 24 hours of the day. Creating a circle, then splitting it into six segments (one for each lane of the moror way), then using the number of cars to plot points which form the “blob” is an attractive idea too. 

Fortune 500 Catalogtree: 

This piece shows companies, each with its revenue, and profit or loss neatly noted in Catalogtrees rendition of the 2008 Fortune 500 list. The energy of commerce almost takes on the radiance of a sun, as all the companies steer towards the gravitational centre of the image. 

Im liking the idea of representing the “energy” of the data, with everything being drawn towards the cenre of the image. 

Similar Diversity: 

In this piece, created by Strukt Design Studio, words from the major world religion (Christianity, Islam, Hinduism, Buddhism, and Judaism) are abstracted, and rendered devoid of moral and emotional judgement. This enables this piece to show connected the religions actually are. 

Although unrelated to my project, i like the way the data show here has been visualised.

Crystallography - Mineralogy, by Ondrej Jób: 

This extensive work deliberately blurs the boarders between art and analysis. Using varied and diffuse techniques of subjective and objective analysis, the inner nature and form of crystals are celebrated. 

Sunlight Calendar, by Lee Byron: 

This is definitely one of my favourite pieces, if not my absolute favourite piece form both books. Created to visually represent the average colour of light in the sky from various places around the world, at specific times. This representation creates “a river of light”. The character of each location is captured simply by gauging the strength of the seasons, and rhythms and changes in vitality wrought by the suns awakening human and natural activity... it makes me wonder what pattern would emerge from the sky above the A13 / South London (the direction my webcam is facing).

Virtual Water Inside Products, by Timm Kekeritz:

Completely unrelated to my project, however i appreciate the way that the amount of water it takes to create basic products has been visualised, almost in a scary way. Who’d of thought that one 500ml bottle of beer takes 150 litres of water to produce (mostly from producing the barley). 

American Toys, by Christopher Baker: 

Using images from the United States Patent Office, the ghostly silhouette of a gun is the residual foot print of all 500 patent applications submitted for the “toy gun” category since the 1800s. The resultant image becomes a collective expression of the prototypical American toy. 

Aleatory Compositions, by Hoagy Houghton:

This is an extract of sheet music written by people who were unaware that they were writing sheet music. Pieces are created by people filling in a blank grid with 7 colours, shading the squares however they liked. A colour scale translated their visual compositions into musical notes, resulting  in a wide range of random chords and abstract melodies. This bears some relevance to my project, as i was toying with the idea of incorporating some sort of sound generation from the 7 lanes of traffic using the A13.

Adaptation (2002) as Data, by Rodrigo Machado: 

Again, unrelated to my project however i like the minimalistic and easy to understand way of showing data. Based on a film called Adaptation, this infographic map shows the length of the film, flash-backs and flash-forwards within the story, the time each character appears on screen, and the relationships between the protagonists. 

Visible Sound, by sounds.butter:

Something i would never do for this project, however i like the idea of the sewing machine being used to visually show sounds. 

Narratives 2.0, by Matthias Dittrich: 

Very cool and attractive form of data visualisation which analyses frequency channels of pieces of music. The program separates the channels, and displays them in a fan like arrangement, and hints at the works rhythm and pacing. 

Black Locust, by Bryan Nash Gill: 

In a classic case of art imitating nature (or is it the other way around?), Black Locust pays homage to a natural precursor of data visualisation: in tree rings, good years and bad are recorded for posterity int he trunks striated cross section”... i believe this is an actual photograph of a trees cross section. An interesting, yet obvious way of looking at the rings inside a tree. 

Murmur Study, by Christopher P. Baker: 

Very cool piece by Baker, comprising of 30 thermal printers which constantly monitor Twitter for tweets including common emotional utterances such as “argh”, “meh”, “grrr”, “oooo”, “ewww” and “hmph”. The tweets are then printed out in an endlessly flowing waterfall of “digital small talk” which accumulates in a tangled mess below, to be forgotten forever... or in this case, recycled. 

SUMMARY: 

Overall i preferred the first book to the second. The first book seemed to me to be more data oriented, where as the second book seemed to be more “art” oriented. Good inspiration from both books, well worth the read. 

[preservedlemons-blog]

Updated Progress:

Since my last post, i have been focusing on my other 3 modules including exams which accounts for the large time gap between posts. 

Since submitting all work needed for the other modules i have: 

- Taken a week off (because i hadn’t had any time to myself since September).

- Done a week of work landscaping in my parents garden. 

These two points set me back, but were absolutely necessary. 

Since then, i have acquired a new Windows desktop PC. Learning the new Windows 10 operating system has been a big learning curve for me, which took some time to adjust to (especially after using OSX for the last 7+ years). 

I have also kept focus on my Creative Project, which is also proving to be a huge learning curve. 

I naively assumed that i would be able to set up my DSLR as a webcam - how wrong i was. I was researching online if this is possible, it turns out that it is (almost) possible using a library called gPhoto ( http://www.gphoto.org/ ). However, DSLR’s are not made for continuous video shooting, they are designed to take photos.. and thus, the only video feed you can get is the “live preview” on the back of the camera which is extremely low resolution. It took me hours to figure this out, and eventually gave in and bought myself a Logitech webcam. 

Not letting myself be defeated, i signed up to the following course on Udemy: 

https://www.udemy.com/deep-learning-for-computer-vision-with-tensor-flow-and-keras/

After following the setup instructions, i quickly realised that my system was simply “too new” to even install the software needed for the course. 

The main issue was that the instructor taught the course using Cuda v9 which runs on Nvidias old series of GPUs (GTX / Titan). However, i could only install Cuda v10, because i have an RTX GPU, this rendered the course un-followable for me, so i got it refunded. 

I have now found an appropriate course which im finding extremely useful in explaining the Deep Learning and Computer Vision concepts i believe to be necessary for my project. I will be completing this course today, which means i can finally begin writing code for my project.

https://www.udemy.com/share/100FTkBUESdFtaQHw=/

A couple of weeks ago i met with Evan Raskob, who suggested that i look into ways of visualising the data gathered from my project (detailed traffic counting of the A13 as it goes through Canning Town). Evan pointed me in the direction of a of book called Data Flow, Visualising Information in Graphic Design by Robert Klanten. After browsing the internet for the book, i discovered there were two versions of it, and coincidentally, both were in Goldsmiths library.

I read through both books, which are made up of images showing how people display data in creative ways, accompanied by a description of the piece. I found both books to be quite eye opening into ways of showing mundane information. I have written a post on the pieces i found inspiring in both books, to see the review, please refer to the post titled “Data Flow - Visualising Information in Graphic Design”.

Evan has also got me thinking about ways of using the live stream from the webcam, to create some sort of generative art piece. I arrived at “the flow” of the road possibly being represented by a “flow field”, and possibly incorporating some kind of generative sound into the piece... although, i have never worked with sound before. 

A thought i had had was that of the road representing a musical stave, each of the 7 lanes representing one of the 7 notes (A, B, C, D, E, F, G).

This is an abstracted idea, and is not set in stone yet. My primary focus at the moment is that of getting the computer vision system set up, and recognising vehicles correctly. Next i want to find a way of inputting this information into a database. After that, i will begin thinking of some artistic flare i can add to my project. 

Overall, i am happy with where im at at the moment, despite the setbacks which i previously mentioned. 

[preservedlemons-blog]

Today i completed the first stage of the course. So far, this course has been an amazing introduction to Computer Vision. 

I have had to comment out lines 15 & 30-32, as my laptop cannot handle the computations to detect a smile.... grumpy laptop

[preservedlemons-blog]

Computer vision

I have been looking into various online courses for OpenCV. This one on Udemy seems to be quite highly rated: https://www.udemy.com/computer-vision-a-z/

[preservedlemons-blog]

Air pollution detector...

I have read the following article on how reliable home made air pollution detectors are:

 http://theconversation.com/can-we-rely-on-diy-air-pollution-sensors-67747

The following paragraph caught my attention: 

“Sensor manufacturers sometimes provide limited information on these low-cost sensors, and it is very easy to use the devices improperly. This is because they are designed to work under very controlled conditions – for example, at fixed temperatures or with limited wind movement – and these requirements often are not communicated to consumers. Measurement accuracy is especially important when we are trying to understand how exposure to air pollutants can lead to health problems. If we rely on poor measurements and reach incorrect conclusions, we will fail to protect public health.“

[preservedlemons-blog]

Air pollution detector...

I’ve been looking into home made air pollution detectors. I found a list of items i could use on the following Instructables 

post: https://www.instructables.com/id/Air-Pollution-Detector/

This uses an Arduino with 6 sensors to detect various poisonous gases. 

[preservedlemons-blog]

Week 5...

This week i have been looking into resources i could use for the potential OpenCV project. Last week Simon recommended me to look at the work of a past student, Joe McAlister. 

McAlisters project, Citybreathe, uses data found by the Environmental Research Group of Kings College London and allows users to check the pollution levels in their area for free. https://citybreathe.com/

If i get a camera correctly counting traffic, i could then start moving onto measuring the pollution levels coming off the road below my flat. 

[preservedlemons-blog]

Thoughts...

I live on the 12th & 13th floor of a block of flats in Canning Town (one of the most polluted areas of London), right above the A13. I have 6 lanes of traffic all day and night going past my flat. Its pretty disgusting. Ever since moving into the flat 2 years ago, i have wanted to know how many cars use the stretch of motorway below my flat in an average 24 hour period. 

Last Friday (19th of Oct), i met up with Simon to discuss ideas for my final project. The outcome of the meeting was basically i can take one of two paths... Go with a light hearted, interactive game / project involving Open CV and Machine Learning... or go with an informative tool to gather data about living in London (measuring traffic flow, sound / light / car pollution levels)... and then maybe measure the impact of these on your personal health and sleep cycles. 

Both of these areas interest me, however at the moment i am leaning more towards the data gathering side... purely because i just want to know how many cars use that road on a daily basis!

[preservedlemons-blog]

View this post on Instagram

A post shared by New Media Art (@new_media_art)

I like this idea of mapping sounds of a choir to the expression on your face... as if you were making the sounds yourself.

[preservedlemons-blog]

Random note...

A thought popped into my head a few days ago (steady on), where i could make an interactive safe, where the user needs to unlock it with their hands to win a prize. This would involve machine learning and Open CV. 

This is just a random thought, not a 100% “i’m going to do this”, however i do like this idea.

[preservedlemons-blog]

I really like this idea of breaking the boundaries of the computer screen using your hands.