Task:

Create an artistic or practical interface that utilizes KNN Classification. Be as experimental as possible with the real-time training feature. Avoid that random inputs trigger random effects. 

Final Product:

Process:

I started by experimenting with the in class KNN code, I realized that aside from the cool effect that it creates, it can be a great tool for recognizing objects. 

By taking a snapshot of a still background, we can now trace any moving figure that is different from the original rgb values of each specific pixel value.

A clear human figure can be seen as I walk around the canvas.

I decided that this would be a great application for the generation of a greenscreen effect.

improvements from inclass code:

I implemented a training portion to the code, unlike the snapshot that just takes one frame. The code now takes in real time footage until the average difference is lower than a threshold value, when most pixel value stays still and constant, the training will be finished and the trained model will be stored unless the user presses the space bar to train again. 

Then I filtered out value by pulling values that are close to 0 to 0, and value that are high to 255.

using this filtered difference value, I adjusted the opacity value of the output pixels, making the pixels that are very different to be completely transparent, revealing the realtime camera footage underneath, while pixels that are the same as the trained model will be changed to green.

video

the one disadvantage to this approach is that if the color of the moving object is close to the still background, there will be extra green on the moving object.

If this program is ran in a properly lit room, with clear differences in color between objects, then it will work great.

Overview

For this week’s assignment, I utilized a KNN model to develop an interactive musical program. Users can trigger different songs from various musicals with different body gesture. For instance, when they cover half of their face, the music playing would change to the phantom of opera; when they pose their hands like a cat’s claw, the music would change to “Memory” from cats, when they wear anything green, the song would change to “Somewhere over the rainbow” from the wizard of Oz. The model I used also allows users to train their own dataset, which improves the accuracy of the output.

Demo

Technical Problem

At first I want to use both the users’ speech and movement as input to trigger different output, however, I had some difficulties combining two models together. Still, I think it would be more cool if the user can use both their songs and dance to interact with the musicals.

This week, I used KNN classifier to developed the classical snake game based on webcam image matches with pre-recorded gestures with four classifiers. This image match could be further used to involve in other games that requires multiple inputs. This assignment would start with taking four snapshots, each matches one direction (left, right, up and down) that the snake would go. Then the snake game would start, where the user would used gestures to control the movement of the snake.

Continue reading “MLNI-Innovative Interface with KNN (Wei Wang)”

This week I want to use KNN and poseNet to make a filter, because I want to make some filters for users in my final project. I made three themes for the filters. One is Christmas theme. If the user make a decided pose, then there will be flowing snow, red hat and white beard on the screen. For the Halloween theme, I made a spider and a pumpkin head. For the Spring Festival I added the fireworks and lantern.

At first, I want to change the RGB of the canvas. For example, if the classification result is A, everything in the canvas seems yellow. If the result is B, the color will be blue. If C, then the color will be red. However, it didn’t work. That was because I just directly used the code from last class and it became a screenshot so the canvas was still. Then I made some changes but it couldn’t work either. But it provided me an idea for my final project. If I solve this problem, I can apply it on my final project.

Sound Password

This is a mini project with real-time trained “machine learning model” where users will use sound to train the model. Users need to record a sound clip as password in order to save some secret information. They can unlock the information by reproducing the same sound.

Implementation

In this assignment, I utilized the FFT algorithm provided by p5 library. After processed by FFT algorithm, the sound characteristics at a frame is represented by a number array of length 1024 with each entrance a value ranging from 0 to 255. To store the sound data in a certain period of time, I used a 2D array.

The KNN methodology used here is comparing the difference between two number array. Iterate through two arrays and compute the absolute difference between two numbers with the same index. Finally, divide the sum by 1024 and the derived value is the difference of two sound waves at the same time point.

Then we can expand this technique. Compute the accumulation of such difference and return the average value in each time point. The final value we get is a value ranging from 0 to 255. However, after several test, I found this value quite low and decided to use 10 as threshold.

Difficulties

After finishing the assignment, I was still unable to synchronize the two sound clips well. A user may successfully reproduce the same sound clip as the recorded password. However, they may not start at the exactly same point. Therefore, such offset causes obvious inaccuracy in the final difference.

I think the algorithm I used in this project is quite weak. To make the algorithm robust, I believe some mathematical techniques need to be involved.