MLNI week2 case study

 Visual Recognition–OrCam

Shenshen Lei(sl6899)

Visual Recognition:

 Visual Recognition is one of the widely employed AI technology today. There are plenty of Visual AI such as facial recognition, image search, color recognition, etc. Visual Recognition promotes the development of Zero UI. Our group found a typical and very useful product that based on Visual Recognition technology: OrCam.
    OrCam is a company developed a series of products that help those who are blind or visually impaired. OrCam MyEye could recognize the text and faces at a certain distance in front of the user and then read out the content to them. It could also recognize some body language such as “looking at the watch” and tell the user what time it is. It improves the reading efficient of the users and makes their lives more convenient.
    However, we believe that the product could be improved in several aspects. First, the way to trigger the text recognition function is to point at the word, or the cam will start to read the whole page of the text. Blind people may not know where the word is, so pointing-to-start sometimes does not work. Another thing is that if the cam could trace users’ pupil when they look at the text, users can get simultaneous feedback without using their finger to point texts. Also, the recognition range of the cam is too limited. If it can enlarge the distance of recognition it can help in more aspect such as recognize the traffic light or warning the user earlier if there are barriers. Finally, the OrCam may also change the design of the outlook.

MLNI Week 2 Rain Sketch (Jessica)

Link: https://editor.p5js.org/chonjessica/full/fgyOHRBbj

For this homework assignment, I decided to make a simple rain drawing with spinning raindrops. 

I didn’t have too many issues while doing this assignment. The beginning was a little difficult because it’s been a while since I’ve used p5 but going over how to use it in class was helpful. I had to google what some lines of code meant such as scale() and how to use rotate() but I figured out what to write after looking on the p5 website.

However, I did have trouble figuring out why my raindrops weren’t being drawn initially, but remembered that the code is read and run through from top to bottom. Initially, I had my code drawing the ellipse for the raindrops first, then all the following transformation function and display functions. But once I placed ellipse() last, multiple raindrops were drawn. 

In the future if I were to return to this, I would have the raindrops falling so that it would be more realistic, rather than them just spin in place. I would also try and draw them into a more raindrop-like shape. 

Week 2: Introduction to Machine Learning Models and ml5.js

Semi-Conductor

Partner: Jessica Chon

What the project is:

Semi-Conductor is a virtual conducting app where arm movements yield a reaction from your virtual orchestra. It is quite fun, but there are definitely bugs, like sometimes you have to amp up for energy because some subtle movements work and get registered with the system while others yield no reaction with the virtual orchestra (Which is sad and the slightest bit annoying). 

Beneficial Qualities

It goes beyond say Leap motion plus because it has more range in motion, that is even slight changes to position and angle allow for a completely different reading and hence reaction from the software. 

It makes the user fully interact with the technology, which makes it “Zero-UI”. So yeah, if you get your whole body in the movement then it can yield a very clear, long response from the system which is really cool. I guess in that way it is kind of like Kinect, but just off of your webcam which makes it far more accessible than Kinect based programs. 

How we would implement the tech:

Gauge the effectiveness of young conductors (conductors in training). You could train the virtual orchestra based on say a real orchestra to improve the receptiveness of the program to the movements and make it as realistic as possible. 

Helpful for deaf students because you can still see the metronome and instruments moving and having a sense of control keeping tempo because of visual cues. Music is a requirement in a lot of schools, maybe this is a way to engage a population that previously just couldn’t participate because music was so reliant on one of their senses that they just didn’t have or severely lacked in.

Other simulators such as sports games/training, presentations. It would be cool to use this technology to train goal keepers in soccer, or like even batters in baseball. There is only so many situations that they can train for, but like the computer could test extreme cases to even prep them for unlikely, but match-winning potential goals/ pitches.

Our presentation 

And then… my code for the P5 practice using transformation and display functions can be found at this link. 

Hopefully this works!!! “(Maybe not, high key not sure about the whole zip-folder thing).

Week 2 MLNI – Partner Presentation (Jessica)

Project: Semi-Conductor

Link: https://semiconductor.withgoogle.com/

Partner: Eszter Vigh

Summary

The project is a musical conductor simulator where the way you conduct affects how the “orchestra” plays. The fast you move your arms, the fast the music plays. You can also control the volume by either crouching or raising your body. Lastly, you’re able to control which side of the orchestra you want to play based on you turning left or right. 

Beneficial Aspects

You’re able to use your whole body as the control, and several aspects of it such as position on the screen and how your body is turned, not just arm movement. This helps make the experience more immersive and is another example of technology moving towards “Zero-UI”.

Technology Applied

Eszter and I discussed the possibilities of how this technology can  be used and one of the ideas we talked about was how this could be used not just as a game, but also as a way to help deaf students learn how to conduct, by training their sense of rhythm since there is a more visual cues based on a users actions. Music aside, we believe this technology could also be used as part of a sports-related game or training, and part of presentations in large lecture rooms.