Final Project – Christmas Face Filters

Christmas Face Filters

p5.js sketch link: https://editor.p5js.org/cs6240/sketches/Hbc23sJtx

My project will present five different Christmas face filters to the user, using p5.js and HandPose models.

Summary

Using two machine learning models, HandPose and PoseNet, this project will detect key facial parts and apply selfie filters to the user. The HandPose will be used for the user to select the type among five filters by moving their hand in the air instead of clicking the button or touching the screen like other selfie apps. More specifically, the position of the index finger will determine the filter: each of the five boxes represents a certain filter and the user can point one of these boxes to change from filter to filter. If the index finger isn’t on any of the boxes, there will be no filter and the canvas will show the normal self. However, if there are two hands detected on a screen or the hand is cut off the screen, it might confuse the detection of the index finger and become inaccurate. The PoseNet will be used to find the user’s nose when they are standing in front of the camera. Then, it will attach features of the filters in relation to the coordinates of the nose. For instance, when the PoseNet identifies the nose, it will add Santa’s white mustache below where the nose is. It will offer five face filters – Santa, snowman, Rudolf, elf, and polar bear – in the theme of Christmas to celebrate the upcoming holiday. It is geared towards offering a fun, interactive experience to the user, in which they can become a Christmasy person or animal.

Inspiration

I initially got inspired when I was playing around with Instagram’s countless face filters. I liked how it can accurately identify my facial features and add images such as a cat’s ears accordingly. This reminded me of one of my past assignments that used poseNet to find the nose and put a Santa filter on the user. I decided to develop from here by adding more types of face filters and combining it with a HandPose model for more interactivity.

Process

Before I began coding, I collected visual assets from freepik as it presents countless free-licensed images. I gathered cute clip arts that I need to upload on my canvas and made a few adjustments using Adobe Illustrator such as re-scaling and simplifying the image.

Then, I started coding by uploading two machine learning models.

Then, I assigned variables to the coordinates of the nose and the index finger so that the value of the variable determines the type and where to apply the face filter.

Assigning variables to the coordinates of the nose

Assigning variables to the coordinates of the index finger

Then, I created five if statements for the five boxes on the canvas. If x and y coordinates are within a certain range, it will output a certain image. All the values were calculated by the width and height of the box.

Audience

The target audience for my project is people of all ages, sex, and race (even Santa can try it!). Anyone who wants to join in this Christmas spirit is welcomed to try my project. It is likely that the younger demographic find it more appealing as the face filters are something that they are used to, have fun with, and are easily exposed to. On top of that, the new interactive feature of allowing the user to pick a filter by pointing in the air will potentially bring in more teenagers and young adults.

User Testing and Development

During the process, I did user testing with my friends. One of them commented that the experience will be more entertaining if it has more decorations around the canvas, so I added a small Christmas ornament at the top of the canvas to fit the theme. Another commented that it is quite difficult to aim the box correctly so I added one sentence instruction at the bottom of the canvas to make the user experience easier. Generally, they liked how my project is interactive whilst offering various types of cute face filters.

Next Steps

The next step would be perhaps replacing the models with MediaPipe to see if the computer lags less and detects the key coordinates more accurately. Another would be flipping the canvas horizontally so that the canvas appears like a mirror, which is what we are more friendly with. Lastly, I would like to add a camera function to allow the user to screen capture the canvas and download the image like the real selfie apps.

Link to Presentation Slides:

https://docs.google.com/presentation/d/1KcoS1WRJMON72fcNSJLF2pez6sYK3KXfQzloPPf4mW4/edit#slide=id.gf7e999be9d_0_854

p5.js sketch link: https://editor.p5js.org/cs6240/sketches/Hbc23sJtx

Final Outcome: