Author: Wenhe Li

Intro

ML5 is a user-friendly machine learning framework that is for users who are new to ml field. It covers easy-to-use models and examples.  Among these models, I want to point out the BodyPix model.

What it is?

BodyPix can do segmentation on human body pictures. Instead of simply cutting the figure out of the overall picture, it can also detect every single part of the human body like the arms, legs, and head (it can even tell right and left!). 

Thus, this model can be a good replacement with Kinect. We can use it to track the user’s hand or head positions.

In addition, this model can track multi-people in one scene, which means that it can fire some better performance in terms of complicated and multi-users gesture detection.

Above is a demo of the BodyPix model where you can see that it detects the different parts of the human body and colored them separately. 

Potential Usecase

One thing comes into my mind is like the smart danmuku like Bilibili and Netflix where the comments or subtitle will not block characters in the scene.

Also, this cloud serve as a replacement of Kinect as I have mentioned before. 

Case study: Alibaba – ImgCook 

Motivation:

The aim of developing Imgcook is largely due to the heavy business code (duplicate view development). Honestly, most of the requirement and codes are redundant work which stimulates the demanding of directly generating codes based on the screenshot to free the labor that invests in redundant work.

Approaches:

In terms of the approaches, the Imgcook works in a way where transfers the sketch/screenshot to the middle structure and maps the middle structure into certain code paradigm like React or Vue.

Thus, the core part is to recognize components and patterns from a sketch and translates them into the middle representation. In this part, a bunch of technologies like OpenCV, edge detection, SSD and RNN are applied to ensure the outcome.

Current:

Right now, the Imgcook has involved in real production. And it has dramatically increased the efficiency of converting the sketch to real code.  Normally, a sketch that needs 6 hours to implement will be reduced to 1 or .5 hour. And they are trying to continuously develop it comprehensively for more functionalities like a logic generation and customized code style.