Name

What If U Re Beauty (BeautyMirror)

GitHub

https://github.com/NHibiki-NYU/AIArts/tree/master/Final

Source

Proposal | Paper | Pretrained Model | Presentation

Tech Specs

In this final project, I mainly separate the work into two parts:

• backend: to be in charge of computational jobs and static servers.
• frontend: to interact with the user, take the picture, send it back to the server, and display the result of the GAN network.

So, the pipeline of the whole steps are:

1. The frontend takes a photo by webCam
2. The frontend process and compress the image at frontend and post it back to backend
3. The python server receives the image and transfers it to numpy matrix
4. The python server passes the matrix as input to TensorFlow backend and gets the result
5. Again, the backend server transfers the output matrix to image and encode it to base64 string as a result for frontend request
6. The frontend gets the base64 image and displays it on canvas

Development process

Firstly, I used a method navigator.mediaDevices.getUserMedia to create a context to activate WebCam. Then, I create a <video /> element and set its source to the context. The realtime camera should be displayed on the screen.

After that, by using canvasContext.drawImage(video, x, y, w, h);, the video frame could be extracted to the canvas. When the user clicks, we only need to pause the update of the canvas and send the image on canvas to the backend server.

Obstacles

There are many problems when I want to publish the demo to the world-wide-web.

Unable to connect to HPC

Since HPC in NYU Shanghai does not have a static public IP address, it is not possible to access the machine directly from the outside part of the NYU network. So, I need to find a way to break this NAT environment. There is an open-sourced software called ngrok, which allows users to run client mode on their NAT server and run server mode on their host server which has a public IP bind on it. When the user requests the IP address via the host server, the host server will connect the NAT server through the tunnel. The users, therefore, have access to content on the NAT server without a public IP.

Link To: ngrok

WebCam only be allowed on https

Another obstacle is the WebCam can only be launched on https (Http over TLS). So, I use letsencrypt to issue a certificate for a domain. (I used my own domain in this scenario, but we can also use those public domain services like nip.io or xip.io)

When got certificates and keys from letsencrypt, we need the file privkey.pem for the private key and fullchain.pem for the certificate. The flask server can be start like this

Now the server should be started at :3003 with https encrypted.

Name

BeautyMirror

Source

Paper: http://liusi-group.com/pdf/BeautyGAN-camera-ready_2.pdf

Pretrained Module: https://drive.google.com/drive/folders/1pgVqnF2-rnOxcUQ3SO4JwHUFTdiSe5t9

Presentation: https://docs.google.com/presentation/d/1DKCaDpAfye6AF4DmrT1d3hjaJCn3PrIQfMcUBAmh2-A/edit?usp=sharing

Inspirations

Nowadays, there are lots of beauty cams in the market that can beautify user’s portraits (transfer the style of makeups) and outputs a better-looking photo than the original one. However, users can either have limited options to select preset models or manually set some abstract configurations described by jargon. These configs indeed guarantee the quality of outcomes, and yet they limit the creativity. The face can be a very personal thing and should be customized based on users’ will.

So, here comes the BeautyMirror, a GAN network by Tsinghua University that extracts makeup features from a face in one image as a model and transfer the makeup style of the input image. The network will detect some feature points of faces, for example, the style of the nose, lip, eye shadow, etc.

The advantage of the project is that by utilizing the power of GAN, the users can change their face style to any portraits they upload which gives the users more freedom when experiencing the project. The challenges are also lying on the ground that I need to be cautious when selecting preset models(images) that have an impressive impact on users.

Demo