Please find examples of image works (stills, videos, even audio) that are based on using more than one image taken from the exact same spot.
What I found:
I watched this really interesting music video on youtube where the whole song was about time. The video was completely done through a normal camera, no green screen or computer graphics was involved. Basically the music video shows a time lapse of the same spot to create an effect of time changing, while the artist lip syncs to feel like he is singing at a really slow pace. So that when the video is sped up, it seems like the singer is singing at a normal time while the passengers and cars are moving at a really fast pace. I think this example is great because it demonstrates a powerful effect that is only viable when everything stays in the same place. Also that it is something that could be achieved without complex computer graphics or other kinds of post filming manipulation aside from speeding up the video.
My first impression of this technology is that it is definitely achievable with the technologies that are already available to us, someone just had to do the work of actually putting together the program that has the power to create 3d visualizations of 2 dimensional images. But after I found out that this project was done in 2007 I am still very surprised that it was something that was developed so early. The advantages and actual applications for something like Photosynth is limitless. The example they used was ironically the 3d reconstruction of the Notre dame using only photos taken by tourists that was posted online and tagged with Notre dame. This was not as cool as it was back in 2007 as it is now, due to the Notre dame just being burnt down recently from a fire. Even if we had the technology to digitally document the architecture with 3d scanners or drones, it will still be impossible to restore something that is already destroyed. making Photosynth the only possibility to reconstruct the Notre dame. The specific application of this technology in the construction of non existing objects is only achievable through the usage of machine learning along with photo merging algorithms.
After checking the recent activities of the original team that worked on Photosynth, most members are still working in the field of machine learning. But not everyone is still in the specific subject of spatializing 2d images because their are a lot of problems when it comes to this technology. The source of image data for the creation of this 3d object are usually not trustworthy. Images could be photoshoped/manipulated and as one of the examples that the developer brought up during the ted talk was that one of the data points that the system picked up was actually a image of a poster of Notre dame as opposed to the real thing. By constructing something 3d with limited 2d data, it is hard to get an accurate picture of the whole thing. We can get a rough outline but there are also pieces that are missing to the greater puzzle. Therefore this technology can not be used as a trustworthy program if we aim to recreate an accurate depiction of a 3d object, with data missing some parts of the object has to be either left blank or imagined.
I am very exited to what this technology is capable of, I am especially interested in the part that the technology is capable of piecing fragments of 2d images for the construction of something in 3d. I can see its advantages in the usage of preserving historical sites as well as a strong tool in creating something that no longer exists or never existed. Such as creating a 3d representation of a building that only exists in a comic book. As long as there are enough 2d information of something, it could be pieced together even if it does not physically exist.
Create a generative and interactive portraiture that utilizes U-NET or BodyPix. The sketch needs to demonstrate proper use of pixel iteration and manipulation as well.
Final Product:
Process:
I first started by checking the individual parts that body pix was able to detect by coloring different parts using different colors
After that I decided to try and implement a clear contrast between the body and the background by changing the size of the ellipse drawn.
The next step I took was adding a light effect by calculating the distance between a certain coordinate to the position of the ellipse. Changing the rgb value of the ellipse fill.
I made this effect look even more like a light by decreasing the size of the ellipse as well as setting both red and blue to zero as it approaches max distance.
The final step I took was importing the gravitational orb code that I created for the OOP assignment and using that as the source of light.
Final Code:
console.log('ml5 version:', ml5.version);
var lastLoc = []; var array = []; var maxN = 100000; var counter = 0;
let bodypix; let cam; let segmentation; let test; const options = { outputStride: 8, // 8, 16, or 32, default is 16 segmentationThreshold: 0.3, // 0 - 1, defaults to 0.5 }
function setup() { createCanvas(640, 480); array[0] = new Spark(1,200,0,1,0); cam = createCapture(cam); cam.size(320, 240); cam.hide(); bodypix = ml5.bodyPix(cam, modelReady);
//test = createVideo('hi.MOV'); //test.hide(); }
function draw() { background(0); //test.loadPixels(); for (var i = 0;i < array.length;i +=1){ if (array[i].alive()){ array[i].gravity(); array[i].motion(); array[i].show(); }
if (segmentation !== undefined) { let w = segmentation.raw.width; let h = segmentation.raw.height; let data = segmentation.raw.data;
let gridSize = 5; fill(255,0,255); for (let y = 0; y < h; y+=gridSize) { for (let x = 0; x < w; x+=gridSize) { let index = x + y*w; // *** let mappedX = map(x, 0, w, 0, width); let mappedY = map(y, 0, h, 0, height);
let index2 = (x+y*width)*4;
if (data[index] == -1){
//textSize(10); //text('.',mappedX, mappedY) } else if (data[index] >= 0){ noStroke(); let rvalue = (abs(dist(mappedX,mappedY,array[0].getposx(),array[0].getposy())))*1.2; let bvalue = 255; if (rvalue < 0){ rvalue=0; bvalue = 0; } else if(rvalue>255){ rvalue=255; }
let size = map(rvalue,0,200,10,1); if (size < 0){ size=0; } //console.log(size);
I am very excited about the new technologies that Facebook have been working on with their VR department, especially with their developments in hand tracking and brain reading accuracy. Even though to create a fully functional brain reading interface will not be achieved anytime soon, Facebook have teamed up with ctrl labs who have already developed a wristband that can interpret brain activities. I believe that neural interfaces will be a very important technology in the future, as it opens up limitless possibilities for the application of this interface. I definitely agree with Facebook valuing the potentials of a neural interface and spending their resources on the development of this technology, but I disagree with the focus on wristbands. Because I do not believe the wristband can capture enough information to perform tasks that we envision neural interfaces to be able to do.
Hand Tracking:
Hand tracking was the other big thing that was announced in the developer’s conference. I am sure that machine learning and computer vision was capable of doing hand tracking a long time ago, but I am still very surprised by the accuracy and precision of Facebooks demo. The video shown was so accurate that it could be the level of precision one would expect from a glove with sensors on it. Not only did their team achieve a high precision network, they also cut the cost/computing resource/battery cost by perfecting their code and training data. Instead of using multiple cameras or depth cameras, they ended up using 4 monochrome cameras on each edge of the oculus and was able to sense the small movements of a hand. There is one disadvantage about hand tracking and that would be the lack of feel or haptics when interacting with objects, one of the comments on youtube was that without the handle it would not feel like holding an actual sword anymore. I totally agree on the disadvantages of this lack in haptics, though I believe that the advantages outweighs the disadvantages by a lot.
Spatialized Music:
I am really interested in audio so I decided to check out their presentation on spatialized music. It was really cool to learn about actual practices in the field of creating music for VR. Techniques such as quad mixing, ambisonic mixing was all completely new concepts to me. It was also really cool to learn about the workflow of creating a soundtrack for a VR based experience.
Part2 : images, videos, or audios taken in Shanghai that have absolutely no signs of humans
I took 8 pictures of nature without signs of humans, it was really hard to get absolutely no sign of human in the picture so I had to crop out certain parts of the image so that no buildings are shown.
aside from these pictures I also have recording of ambient sounds on my phone that I recorded at Minsheng road over the summer. I think this is really suitable for what we are doing because it captures the sounds of cicadas and other animals/insects sounds.
Create a screen-based interactive game that utilizes “abstract” buttons and poseNet. The sketch needs to demonstrate proper use of object-oriented programming concept as well.
Inspiration:
I wanted to create a space shooter/space invader type game where the player controls the spaceship which shoots bullets to hit targets.
I plan on utilizing Posenet by using face coordinates to control the space ship.
Final Product:
Progress:
I started off by creating a spaceship, as soon as Posenet detects a face it will use the nose coordinates to create a ellipse that fires particles. which is defined as a class named particle()
Then I made it more game like by adding a new class named enemy(). By shooting the enemy with particles, they will be removed from the game and score will go up.
I then added the ability to fire at an angle by tilting your head. I achieved this through calculating the difference between the y value of the left and right eye, then I smooth out the difference value by dividing so that it is not sensitive. The player should be able to shoot straight upwards without keeping their head perfectly still.
The last change I made to the game is adding an “abstract button”, I decided to change the game so that particles would only be fired when the right hand is held upwards.
Reflection:
I am very surprised by how much interaction and immersion posenet can bring to a simple game by setting the controls to human movement as opposed to keyboard/mouse interaction. I ran into a lot of problems when creating this project, I wanted to add sound and sprites to the project to make it look more polished, but my computer/browser does not load things properly. The macbook’s camera also turns off from time to time making debugging difficult. The mechanic of firing with hand position is also very glitchy cause it either gets in the way of face detection, or it goes outside of the frame captured by the camera. Aside from the problems, I think just with what I have right now it is already a game that is fun to play. I believe by adding sound/sprites and more levels/power-ups/types of enemies, this can be a really fun game.
Final Code:
console.log("ml - ",ml5.version); let particles = []; let targets = []; let cam; let img; let poses = []; var timer = 0; let score = 0;
//console.log(particles.length); //particles.push(new Particle(width/2,height/2)); for (let i = 0; i<particles.length;i++ ){ particles[i].move(); for (let i = 0; i<poses.length;i++ ){ //particles[i].checkinteraction(width-poses[i].pose.nose.x,height*3/4); }
particles[i].display(); }
for (let i = particles.length-1; i>=0;i-- ){ particles[i].checkedge(); if (particles[i].isDone){ particles.splice(i,1); } }
//console.log(targets); for (let i = targets.length-1; i>=0;i-- ){ targets[i].checkedge(); targets[i].move(); targets[i].display(); if (targets[i].isDone){ targets.splice(i,1); } }
//check collision for (let i = targets.length-1; i>=0;i-- ){ for (let b = particles.length-1; b>=0;b-- ){ check(targets[i],particles[b]) } } while (particles.length >500){ particles.splice(0,1); //index,num of element }
}
function check(target,bullet){ let distance = dist(target.x,target.y,bullet.x,bullet.y); if (distance <= 17){ target.isDone = true; bullet.isDone = true; score += 100; } }
function mousePressed(){ //particles.push(new Particle(random(width),height-10));
