Week #2 Assignment: ml5.js Experiment — Lishan Qin

Playing with ml5.js example      

This was my first time using the ml5 library. I started with the imageClassifier() example. In my daily life, I’ve seen and used many applications of the image classification/recognition technology, such as Taobao, Google translator, Google image recognition and so on, and I’ve been curious about how it works for some time. Luckily, this ml5 example has shed some light on this question.

Technical part

The ml5.imageClassifier() contains a pre-trained model that can be used to classify different objects, and the ml5 library accesses this model from cloud. The In the example code of the sketch.js, I can see that the first thing the program does is to declare a callback used to initialize the Image Classifier method with MobileNet, and then declare a variable to hold the image that needed to be classify.

After that, it initializes the image classifier method with MobileNet and load image.

It then defines a function that can get the result of the classification or display some of the errors in the program in the console. (I’m not sure if this function counts as a promise though.) The output of this function not only shows the results of the classification but also the confidence on how accurate the classification is.  Finally, there is a function that displays the output of the function using p5.js.

I assume this example project has concluded many machine learning technologies such as deep learning, image processing, object detection and so on. I think the pre-trained model of this example must concerned a great deal of the machine learning technologies. I will try to look into it in the future.

My output:

Questions:

One of the problems I encountered when trying this example that I still don’t know how to solve is that the program always seems to crash when I use an image from my local file. It failed to load the image and often give wrong classification. I still don’t understand why this is happening. The output of the grogram often looks like this when I use an image from my computer instead of online:       

My code:    

Everything seems to be fine when I use an image from online with an url though…

Thought on potential future application:

Even though today there are already many applications of image classification algorithm similar to imageClassifier(), I still believe the great potential of this example haven’t been fully exploited. Most application of this technology focus on shopping or searching, but I believe it can also be used in both the medical area and the art field. For example, maybe this technology can be used to classify different symptoms of patients’ medical report images like CT or ultrasonic image reports in hospital to adds to the development of future AI doctor…Or it can be used to help fight against plagiarism in art field, or other creative art projects like “which picture of Picasso you look like the most”…The imageClassifier() example, along with other outstanding ml5.js example, have showed me the great potential of ml5 in the future in all walks of life.

Week #1 Assignment Research Work — Lishan Qin

presentation Link: https://docs.google.com/presentation/d/1_2rswKp1qMv_yTluGUt5Egbz3Z8np1Km0AJxH_vETlU/edit#slide=id.p 

Research about the application of computer vision: A Film Directed by an AI Director Eclipse

These readings and TED Talks about computer vision reminded me of an animation I watched in the summer produced by Netflix called “Carol and Tuesday”. It mainly talks about a story taken place on Mars 100 years later from today. Since it’s a story taken place in the future, I saw a lot of visionary sci-fi products in it with great relation to computer vision. One of which I found most bold and fascinating is the application of an AI film director. I found this idea to be so bizarre and ambitious so I did a research on it. It turns out that such application already exists today. AIs have begun directing films for many years, some of the films are even not that bad. In 2016, an AI called Eclipse won an award at Cannes Lions Film Festival for its own short film, challenging viewersby asking the questions “can a film made by machine move you?”. 

What is it?

The film put together a different kind of “film crew” comprising A.I. programs including IBM’s Watson, Microsoft’s Ms_Rinna, Affectiva facial recognition software, custom neural art technology and EEG data. Together, they produced the film “Eclipse,” a striking, ethereal music video that looked like a combination of special effects, photography and live-action. The movie is conceived, directed, and edited all by machine. In the behind the scenes video, we can hear the team members explain how they teach the machine to tell a story.

How did the AIs do it?

Emotional theme

How can an AI grasp and decide an emotional theme of the video? The answer is through the analysis of music(BGM). Music is a crucial part of every film. Thus, knowing the emotional intent in each song is of great importance to the AI. By breaking down the song and analyzing each line and tone of the lyrics with the help of IBM Waston and EEG, the AI is able to lock the emotional theme of the music and decide the emotional theme of the music.

Narratives

Another AI program different from IBM Waston and EEG was applied when creating the narratives of the film. As the most important thing that a director do is to answer questions, the programmer team use Mircrosoft’s AI chat box Ms_Rinna to write the narratives by asking it questions and recoding its answers. Ms_Rinna was used to give the narrative direction on everything of the film from characters to settings on set.

Casting

Using the lyrics and guides and the answers provided by Ms_Rinna, the story of the movie was in place. After that comes the casting. AIs are also in charge of the casting by having all of the actors wearing EEG machine and capturing their performance in close up to get facial recognition and ask the machine to align with the emotional theme and choose the actor. The AI ended up choosing the exact same people the team chose.

Shooting

The AIs are also capable of shooting mostly on its own. With the help of MUSE EGG, IBM, WATSON, AFFECTIVA API and PRENAV drones, the AI also shoot the film on its own.

Editing

Finally, there is editing. Equipped with the ability of image detection, motion detection, the AI is able to process a huge amount of shots and give out dozens of cuts. With open-source library, the AI is also capable of applying different visual filters to interpret the director’s vision.

Thoughts

The reason why I find this project so amazing is that it is such an ambitious program that has combined computer version, big data, machine learning, and many other advanced technologies together, to create an actual piece of artwork that is so eye-opening and revolutionary. It showed us the great potential of the application of AIs in future art creation, proved the possibilities multiple AIs working together can achieve, as well as challenged us to rethink the relation between humans workers and AIs working in the field of art. 

Behind scene video of Eclipse:

https://www.youtube.com/watch?v=XZbcxsHb4Y0

Source: 

Eclipse, the world’s first AI produced short film hits the screens at Cannes

https://variety.com/2017/artisans/production/production-workers-ai-1202447872/ 

https://adage.com/creativity/work/anni-mathison-eclipse-behind-scenes/47918?

Week #1 Assignment Case Study Presentation — Lishan Qin

【Link to presentation slides:https://docs.google.com/presentation/d/1HGNBJahn-no-A4c0an4ArfwE15fDsi-m95B06KopOjU/edit?usp=sharing】

Research:

The case I studied is the AI designer “Lubanner” from Alibaba. According to the annual report released by Alibaba, every year on November the eleventh (also known as the “double eleven festival”), about 40 million different posters and banners are called for for various promotion activities on Taobao. Such large amount of workload used to be extremely expensive and time-consuming for the company. However, since 2017, with the help of AI tech and machine learning, the artificial intelligence designer “Lubanner” created by Alibaba has improved the situation to a large extent.

  “Lubanner” is a self-improving artificial intelligence program designed to create a large number of advertising banners and commercial posters in a short time. With the help of big data and the AI tech, the programmers write functions to train “Lubanner” to receive inputs, process the similarities and patterns between different inputs, and give outputs that meet the requirement by imitating these patterns.  There are basically four things the programmers are doing. First, the digitalization of designing. They teach “Lubanner” to use image processing algorithm to identify and cutout the five major parts of each picture: the product, slogan, background, logo and decorations. Secondly, “Lubanner” archive those data. After that, programmers wrote function to teach “Lubanner” imitate those input posters and banners and create its own work. Finally, it evaluates its output before giving out the final design result. Here is a demo video presenting how “Lubanner” work in action.

The efficiency, low cost, and diversified outputs of “Lubanner” are all outstanding. According to Alibaba, it used to take a junior designer 20 minutes to work out a product banner. However, today, “Lubanner” is able to design up to 8000 different banners with various themes in 1 second. What’s more, the aesthetic of its works is also getting better and better, meaning that its works are getting more and more appealing to customers, sometimes even more so than works created by real humans. The reason why the machine is able to tell aesthetic is that it keeps updating itself by receiving and memorizing the most appreciated data about certain work and working out the key factors. It is clear that most commonly-used apps all have similar UIs, so are many banners and posters. AI designers can abstract and learn from these patterns by imitating them to create more appealing works for customers on their own. Thus, in the future, it’s only naturual that AI designer is going to take over more and more junior design work.

Even though it is clear that AI designers like “Lubanners” aren’t really producing any innovative art works,it’s actually more like a design factory that is able to produce diversified products. Still, these kinds of seemingly soulless yet effective design products have brought huge convenience to business company and are also becoming more and more well-received by the audience. The potential of the AI designer is numerous, and ot will most certainly one day revolutionize the way we define the word “design”.

source :https://www.alibabagroup.com/cn/ir/home; https://medium.com/@alitech_2017