#modelready

[theogoff]

priv link to the output result of the ML code and a draft of some sort of more finalised video project.

For this project I was interested in using the PoseNet model to create a dynamic process of analysing and merging videos with images that would serve as a creative tool for dancers and performers. Similarly, this idea would lend itself to engaging video works where the process could be used alongside any videos where ml5.js is able to pick up body positions.

THE PROCESS

part one - analyseStills

The process took on several computational and non-computational steps that all end up working together in the final process. I first had to develop a piece of code that could take a jpeg and load ml5’s PoseNet to detect positions on still images:

function modelReady(){

  // when the model is ready, run the singlePose() function

  // if/when  pose is detected, poseNet.on('pose', ...) will be listening for the detection results

  // in the draw() loop, if there is any poses, then carry out the draw commands

  poseNet.singlePose(loadedImage, gotPoses);

}

Once the program could detect and store the coordinates of a single pose into the inputs array I was able to iterate through several stills at once for their coordinate data to be pushed into the array that would eventually be referenced in the second piece of code and compared against.

As I wanted my input data to be many different stills from a very similar video to the one I would be comparing them against, I needed to pull lots of jpegs from an mp4 file – at this point I was still using some found footage online of a man dancing with a static camera setup and his full body in shot. To do this I used the command line software ffmpeg to pull a folder of stills from the test video I had been using. This was the command written into the terminal[ii]:

specifying the:

time in/time out

 the video file I wanted to use

The quality (2 being the highest)

The number of frames to pull each second (this is an area of experimentation as there are advantages and disadvantages to pulling a large number of stills)

The file path and filenames of the outputted stills (%4d specifying the 4 leading zeros to organise the jpegs – 0000)

I then created a set of functions that alternated between taking the keypoints from an image, pushing them into an array, and loading in and preparing then next still, linking it up with PoseNet. It was difficult working out how to reference and name the stills correctly but by using a technique by Seaux[iii] I was able to take the correct still at the right time and give it a name that in the second program would be used to compare its keypoint coordinates.

function getImage(currentImage) {

  currentImageString = `scene1outfit1_${('0000'+currentImage).slice(-4)}`;

  // (adjusted to pad number with leading zeros)

  return createImg(`assets/scene1outfit1_output3/scene1outfit1_${('0000'+currentImage).slice(-4)}.jpg`);

}

part two: concept and filming

Now I had the framework for the first half of the program in place, I needed to then capture the footage that would be used for the final piece. This involved working with movement artist and dancer Isabella Pinto, filming some static shots that drew influence from the DIY style dance videos[v] that are common on YouTube and TikTok.

The footage was then processed in the first program, and I had a JSON file for each video that could be used in the second program to the data comparison. There were some adjustments needed for naming the arrays in the objects and the resolutions of the images to be optimal for the second stage.

part three: compareToVideo

The second program needed to play one of the videos and run the PoseNet model on it for each frame so that every frame, this data could be compared to a JSON file of the stills from one of the other videos (which are all preloaded in in setup()). Four functions, findPositions(), getCurrentFrameposition(), getFrameJSONDistance() and findLowestDistance(), as you can see in the code, deal with the comparisons. Initially I used the dist() function that is within the p5.js library but calculating each position using the powers of each keyframe coordinate served to be a more accurate and quicker operation.

In the console, you can see that every frame, each frame distance is pushed into an object and the minimum still is printed with the corresponding value. These are then used to print the correct still into the screen. The value was found using some code from Šime Vidas[vi]

OUTCOME

The final video outcome is a choppy sequence which is a mix of the input video and the file of still images. To capture the video I screen recorded the canvas, this is limited in the sense that the computer struggles to deal with running the code and capturing the screen at the same time, so the final video has quite a lot of lag. For the final film I used the scene1outfit1 stills overlayed over the scene1outfit2 video, and for the second part I used the scene2outfit1 stills over scene2outfit2. I then added some distorted background music to represent the music that the dancer is listening to in the headphones and put the original audio with it.

[i] Tyler Henry, “kinemetagraph - process footage”

[ii] Florida Water Daily “Extracting Still Images from Video with FFMPEG”

[iii] https://stackoverflow.com/a/20460414

[iv] https://stackoverflow.com/a/30800715

[v] Maresque Scott, “PUMPED UP KICKS/DUBSTEP”

[vi] https://stackoverflow.com/a/11143045

[solacetheartist]

• My character from the “Rearranging” music video is all ready for his med school debut! 🤓📘🩺 • • #medschoollife #actingtime #modelready #newsongs #rearranging #hydepark #chitown #collegelifecheck https://www.instagram.com/p/CUvKivSvHSL/?utm_medium=tumblr

[aaronhillsworld]

Soooo..... Which one is your favorite color? 😉 Be Confident. Aaron Hill #modelready #vulnerabilityisstrength #nakedtruth #thenakedtruth #transparency #strength #beconfident #confidence #nofear #fearnoone #judgementwelcomed https://www.instagram.com/p/CIPDRN_ldJn/?igshid=9rz489dpsxn6

[loveomhh]

When you have a TWA, it’s all about the texture and cut. ______ Hair: Debra Hare-Bey Private Parlour At OMhh Beauty Oasis, Model: Kitty Doyle @kittisha_doyle, Photo: @alexanderkusak ______ @_omhhredroom @wbmgetmarriedin @worldbridemagazine #wbmgetmarriedinbrooklyn #wbm #worldbridemagazine #naturalhair #twa #gorgeous #modelready #coils #combtwist worldbridemagazine (at Brooklyn, New York) https://www.instagram.com/p/B3Yd-oUBCl1/?igshid=1o1pywe6shbpw

[destinationnwonderlandd]

Thick Bitch 💋 - - - - - - #thickthighs #thickthighssavelives #thickbitch #instamodel #curlsforthegirls #curvesfordays #curvesaresexy #modelstatus #modelready

[ugdigital]

#Whutchudoin... got our workout 🏋🏾‍♂️ in today! It’s #transformationseason ... #newday #newlife ... #modelready by summer baby!!! (at LA Fitness - LYNDHURST-CEDAR RD) https://www.instagram.com/p/Bvzobo6hlil/?utm_source=ig_tumblr_share&igshid=1duy4651ge9uz

[drgphotostudiollc]

#ModelReady Models: @mistiegibby and @drking0488 ”Thank you to Mistie and David for a awesome photo session!!! I look forward to working with you again in the near future!!! Photographer: @drgphotostudiollc . . #TheDRGexperience via DRGphotoStudio.com . . #ShopSmall #SmallBusiness #MadeByDRGphotoStudioLLC .. . BOOK your photographer Let's create some #memories... . . #DRGphotoStudioLLC #AtlantaProfessionalPhotographer #photography #onlocation #StreetPhotography #CouplesSession https://www.instagram.com/p/BuUVWM6BPOH/?utm_source=ig_tumblr_share&igshid=1g5pzff10kzv8

[mykegrymz]

BACK AT THE SKATEBOARD PARK PT.2. JUST CHILLING WIT MY HOMEGURL SARA. 😎💯🚬 ▪ ▪ 😉STAY FOCUSED!! KEEP WORKING!! ▪ ▪ 📷🔥@summeradamsphotography. (IDreamAboutPhotography) 👟@vans @vansskate @vansgirls ▪ ▪ 👤@mykegrymz 🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸 📢TOP S/O's. THX FOR YOUR SUPPORT.👊THE REST IN COMMENTS. 😎💪💯 @ ◀◀◀ 😉👏 🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸 ▪ #skateboardpark #modelready #vans (at Kennesaw, Georgia) https://www.instagram.com/p/BsqRrAGA8fQ/?utm_source=ig_tumblr_share&igshid=1qvy5e57o654z

[oshiri-sisters]

First walk of June with my favorite model ☀️☀️☀️#summer #dog #modelready

[ellemechelle]

View this post on Instagram

A post shared by Shop Elle MeChelle (@stylinphotodiva)

The set up is everything!!!!!! #modelready #makeupfirstschool #cultivatingmygreatestlife #photoshoot https://www.instagram.com/p/BuKBO4gFZEd/?utm_source=ig_tumblr_share&igshid=1x5woguuun7up

[approvedtrash]

About the other night 🍩 #krispykreme #donuts #modelready #happymonday #happybirthdaytome (at Krispy Kreme Daytona Beach)

[booboobeckie]

#sunglasses #grandbabiesarethebest #cloudporn #babies #kids #beautiful #gorgeous #justgoshoot #lips #face #prettygirl #hair #precious #reflection #modelready #granddaughter #yellow #nose #windowstothesoul

[debotoppless]

That one time I didn't know I was taking a picture!!!!! #modelready #guapo #whatabeautifulchocolateman #saucey #mygoodside (at San Diego UNITED Training Center)

[loveomhh]

Hair: Debra Hare-Bey Private Parlour At OMhh Beauty Oasis, Model: @d.arnez @_omhhredroom @_omhhinc @carltonjones212 @wbmgetmarriedin @worldbridemagazine @simplyndesign @legacy_beauloni @cassbromfield @brownstonebridal #wbmgetmarriedinbrooklyn #wbm #worldbridemagazine #groom #groomsmen #thesuit #naturalhair #fauxlocs #localternatives #handsome #modelready (at Brooklyn Borough Hall)

[destinationnwonderlandd]

70 Vibes ☮️ - - - - - - #modelready #biracial #blackgirlmagic #retro #aboriginalpride #mirrorselfie #snapchat #selfie #modelready #modelstatus #instamodel

[drgphotostudiollc]

View this post on Instagram

A post shared by 📸Tyrus Riley Sr.📸 (@drgphotostudiollc)

#ModelReady Candice is an amazing aspiring model ready to take on the modeling Industry. . . Model: Candice Shinall @candaceshinall Makeup Artist: Michelle Carter @laurenmartistry Hairstylist: Tricia Bailey @triciabailey1 Photo Credit: DRG Photo Studio, LLC @drgphotostudiollc . . Let’s create #Memories with DRG Photo Studio, LLC “Follow her Journey” Let’s create #Memories with DRG Photo Studio, LLC “Follow my Journey” #DRGphotoStudioLLC #photography #studio #instudio #onlocation #outdoor #corporate #headshots #profolio #children #family #friends #fun #destination #event #ShopSmall #SmallBusiness #AtlantaProfessionalPhotographer #StreetPhotography #couplesession #EngagementSession #Weddings #GraduationSession #PromSession #MiniPhotoSession #MadeByDRGphotoStudioLLC #StreetPhotography #couplesession #EngagementSession #Weddings #GraduationSession #PromSession #MiniPhotoSession #MadeByDRGphotoStudioLLC https://www.instagram.com/p/BupsXOshJQK/?utm_source=ig_tumblr_share&igshid=1glyx8nuuld72