Posted on by Jinzhong Yu

Week 13: What If U Re Beauty(Final Documentation) – Jinzhong

Name

What If U Re Beauty (BeautyMirror)

GitHub

https://github.com/NHibiki-NYU/AIArts/tree/master/Final

Source

Proposal | Paper | Pretrained Model | Presentation

Tech Specs

In this final project, I mainly separate the work into two parts:

  • backend: to be in charge of computational jobs and static servers.
  • frontend: to interact with the user, take the picture, send it back to the server, and display the result of the GAN network.

So, the pipeline of the whole steps are:

  1. The frontend takes a photo by webCam
  2. The frontend process and compress the image at frontend and post it back to backend
  3. The python server receives the image and transfers it to numpy matrix
  4. The python server passes the matrix as input to TensorFlow backend and gets the result
  5. Again, the backend server transfers the output matrix to image and encode it to base64 string as a result for frontend request
  6. The frontend gets the base64 image and displays it on canvas

Development process

Firstly, I used a method navigator.mediaDevices.getUserMedia to create a context to activate WebCam. Then, I create a <video /> element and set its source to the context. The realtime camera should be displayed on the screen.

After that, by using canvasContext.drawImage(video, x, y, w, h);, the video frame could be extracted to the canvas. When the user clicks, we only need to pause the update of the canvas and send the image on canvas to the backend server.

Obstacles

There are many problems when I want to publish the demo to the world-wide-web.

Unable to connect to HPC

Since HPC in NYU Shanghai does not have a static public IP address, it is not possible to access the machine directly from the outside part of the NYU network. So, I need to find a way to break this NAT environment. There is an open-sourced software called ngrok, which allows users to run client mode on their NAT server and run server mode on their host server which has a public IP bind on it. When the user requests the IP address via the host server, the host server will connect the NAT server through the tunnel. The users, therefore, have access to content on the NAT server without a public IP.

Link To: ngrok

WebCam only be allowed on https

Another obstacle is the WebCam can only be launched on https (Http over TLS). So, I use letsencrypt to issue a certificate for a domain. (I used my own domain in this scenario, but we can also use those public domain services like nip.io or xip.io)

When got certificates and keys from letsencrypt, we need the file privkey.pem for the private key and fullchain.pem for the certificate. The flask server can be start like this

Now the server should be started at :3003 with https encrypted.

Screenshots

Leave a Reply