Tag: aiarts02

Intro

ML5 is a user-friendly machine learning framework that is for users who are new to ml field. It covers easy-to-use models and examples.  Among these models, I want to point out the BodyPix model.

What it is?

BodyPix can do segmentation on human body pictures. Instead of simply cutting the figure out of the overall picture, it can also detect every single part of the human body like the arms, legs, and head (it can even tell right and left!). 

Thus, this model can be a good replacement with Kinect. We can use it to track the user’s hand or head positions.

In addition, this model can track multi-people in one scene, which means that it can fire some better performance in terms of complicated and multi-users gesture detection.

Above is a demo of the BodyPix model where you can see that it detects the different parts of the human body and colored them separately. 

Potential Usecase

One thing comes into my mind is like the smart danmuku like Bilibili and Netflix where the comments or subtitle will not block characters in the scene.

Also, this cloud serve as a replacement of Kinect as I have mentioned before. 

For this week’s assignment, I played around with an ml5.js project called BodyPix_Webcam. I was drawn to this particular project because it reminded me a lot of a green screen or the Photo Booth app on Macbooks.

Basically, it detects the presence of a body through your laptop’s webcam. The output is a real-time video of everything covered in black except for the detected bodies. When I was testing it, it worked pretty well when I was by myself with no one else to detect.

I was curious how well it would work with multiple people, so I asked my friend to test it out with me. This didn’t turn out as well, as parts of our faces were covered in black.

This project was interesting to me because of its potential applications. The fact that the bodies were detected in real time would be very useful and could be used to improve a lot of existing green screen systems. In Photo Booth, for example, there are some effects where you can change the background of your photo. However, in order to use these effects, you have to step out of the camera frame first so that the app can detect the difference in the background with and without the person in it in order to detect where the body is. BodyPix_Webcam, however, eliminates this step. If its methodology was applied to Photo Booth, I think it would create a better user experience for users wanting to use the different background effects. However, it would just need to be further trained so that multiple people could be detected more accurately.

Playing with ml5.js example      

This was my first time using the ml5 library. I started with the imageClassifier() example. In my daily life, I’ve seen and used many applications of the image classification/recognition technology, such as Taobao, Google translator, Google image recognition and so on, and I’ve been curious about how it works for some time. Luckily, this ml5 example has shed some light on this question.

Technical part

The ml5.imageClassifier() contains a pre-trained model that can be used to classify different objects, and the ml5 library accesses this model from cloud. The In the example code of the sketch.js, I can see that the first thing the program does is to declare a callback used to initialize the Image Classifier method with MobileNet, and then declare a variable to hold the image that needed to be classify.

After that, it initializes the image classifier method with MobileNet and load image.

It then defines a function that can get the result of the classification or display some of the errors in the program in the console. (I’m not sure if this function counts as a promise though.) The output of this function not only shows the results of the classification but also the confidence on how accurate the classification is.  Finally, there is a function that displays the output of the function using p5.js.

I assume this example project has concluded many machine learning technologies such as deep learning, image processing, object detection and so on. I think the pre-trained model of this example must concerned a great deal of the machine learning technologies. I will try to look into it in the future.

My output:

Questions:

One of the problems I encountered when trying this example that I still don’t know how to solve is that the program always seems to crash when I use an image from my local file. It failed to load the image and often give wrong classification. I still don’t understand why this is happening. The output of the grogram often looks like this when I use an image from my computer instead of online:       

My code:    

Everything seems to be fine when I use an image from online with an url though…

Thought on potential future application:

Even though today there are already many applications of image classification algorithm similar to imageClassifier(), I still believe the great potential of this example haven’t been fully exploited. Most application of this technology focus on shopping or searching, but I believe it can also be used in both the medical area and the art field. For example, maybe this technology can be used to classify different symptoms of patients’ medical report images like CT or ultrasonic image reports in hospital to adds to the development of future AI doctor…Or it can be used to help fight against plagiarism in art field, or other creative art projects like “which picture of Picasso you look like the most”…The imageClassifier() example, along with other outstanding ml5.js example, have showed me the great potential of ml5 in the future in all walks of life.

INTRO

For me, ml5js is not a complete new stuff. During last year, I started to contribute to severial open source project, including ml5js. So, it is my second time to ’embrace’ this elegant library.

ML5.JS is a frontend-machine learning library that helps beginners to get in touch with machine learning rapidly. It is based on tensorflow.js but cutting out most of the mistery parts, making it a simple but useful tool for all people. With this library, you might not want to customize your own models (nodes, layers, tensors, etc.) But it is very easy to load pre-trained models and beautifully serve the intelligence of machine learning simply.

INTEREST

When it comes to my interest in ml5, I will talk about two of my favourate networks — SketchRNN and GAN(in this case, I mean DCGAN). The former one use the creativity of RNN to let machine create sketch step by step. And the latter one using deep convolutional GAN to generate picture on its own.

Although, with the limited computational power of browser, the output does not always perfect:

The first is generated by SketchRNN (what a lovely cat!). And the second one (emmmm), I do not know what it stands for. Maybe its the deeper thought of machines?

CONCLUSION

Introduce machine learning to the web interface is a significant movement and still needs a lot of efforts. Web is the largest entry of the whole internet and the widest gate between human and machines. So, it could be very fancy and meaningful if web can be equipped with high quality machine learning features.