Category: 5. Classify This

For my Classify This project, I decided to make a program that is able to identify the color of a person’s nails. The color on the screen would match the person’s nail color. 

Link to code: https://editor.p5js.org/aysabellegalvez/sketches/ykXZOv4A0

I really like the idea of a machine that is interactive with a person and can recognize something about the person. Some makeup companies (like Sephora) have their own foundation shade-matching technology, and I wanted to do something similar. I decided to do nails, since I have a large collection of nail polish. 

Examples of program: 

Waiting for Nails to Show

Recognizing Black Nails

Recognizing Purple Nail Polish
No Nail Polish On 

An Error in Recognition

One of the main problems I had with this code is with Teachable Machine not accurately guessing the color. I started over and made a completely different Teachable Machine that had more photos in order to make the system more accurate. It’s better than the original, but definitely not completely accurate. 

The other problem I had with coding this was coming up the visuals of the program. At first, I wanted the screen to solely be the color of the nails, but the user would probably rather have a camera showing them. I have a rectangle with no fill and a thick stroke to be the border that matches the color of the nails. It was difficult to find a way to line it up perfectly with the video. As you can see, there is a black border between the video and the color-changing rectangle. If I had more time or was less concerned about functioning code, I would have made the visuals better. 

https://editor.p5js.org/stpaloma23/sketches/_1afOZvbY

For my Classify This project I chose to train my machine to identify a few objects that I had lying around. Once It detects the object, it displays a photo of the item. Here are screenshots of how it works since you don’t have access to my objects: 

The one object that didn’t train well was my charger. I think the reason why is because it’s just a white block with not very identifiable features, unlike my other objects that are very patterned and distinct. For each of my objects, I did about 200-250 samples. Although I did 200 samples of each object, I didn’t train the model to get a 360-degree view of each object. I think this is why my phone charger fell short. In the future, I would probably do close to 400 or 500 samples and ensure I get a 360 view of each object to make sure it’s easily identifiable to the machine.  I found this assignment pretty fun and it was a nice high-level introduction to what machine learning is and how it is applied to creative coding. Previously, I thought machine learning could only be used for really high-tech science stuff, so it’s nice to see that this concept applies to various fields and disciplines in tech. 

My “Classify This” project is a simple game where the user can control a yellow dot, moving it up and down on the canvas by opening or closing their mouth. The aim is to stay within the bounds of the canvas. I created an image model using Google’s TeachableMachine Toolkit. For the model I designated two classes of images: “UP” and “DOWN”. I then added about 200 images of my mouth opened to various degrees for the “UP” class, and my mouth closed for the “DOWN” class. I also sourced stock images from Google of people doing the same expressions to create a bit more diversity in my data set. When training I kept the default settings for batch size and learning rate, but increased the epochs from 50 to 60, meaning each sample was fed through the model 60 times. I found that this small tweak improved the accuracy of my model while taking a bearable amount of time.

The program is reminiscent of the popular mobile indie game Flappy Bird, where the user taps the screen to move a bird that must evade obstacles coming in both above and below from the right side of the screen. I decided that I wanted to create something similar and found that it was more enjoyable to use facial expressions to control the circle. A challenge I faced while creating this program was the dimensions of the obstacles in the game. The height of the incoming obstacles is supposed to vary which makes the game more difficult since it would end if the yellow dot touches a rectangle. Instead of each new rectangle being a new randomized height, the height would change rapidly while on screen.

[Project Link]

I used sound model as the input of my project. Based on the different sound that people make(whistle, talk, knock the laptop with finger nails), the stars will appear randomly inside of the canvas and light up with different sizes and different colors based on the different sound as the input.

Here is the link to my code: https://editor.p5js.org/ys4136/sketches/M1Grpw8p7

Here is the link to the sound model that I built:

https://teachablemachine.withgoogle.com/train/audio/1_L0eYw-h5oyC3WfCtXd_mg268FFfLPeA

Based on the feedback that I got in class, I added in more stars and a moon with a face, so that people can understand the differences between the three sound classes that are used in my code better. I also added in a text message to tell people to clap, whistle, or knock on something, so that people would not be as confused about what to do when first seeing my code. 

Whistle:

Clap:

Knock:

Trying to give users instructions on how to interact with the program.