Kinnected! Midterm Project Update

Why design for kids?

This topic was chosen during the NYU Abu Dhabi Hackathon where we were asked to design an invention for Covid-19. Many innovations were produced during the pandemic for people of different ages, but very few were designed for toddlers (age between 1-4). 

Stage 1: Second-hand research

We conducted a series of second-hand background research on toddlers’ conditions during the quarantine. In one of the researches, researchers surveyed nearly 2000 children in second through sixth grade both in Wuhan, China and in a city about 50 miles away. After an average of 34 days under the lockdown, 23% of the kids reported symptoms of depression. 19% reported symptoms of anxiety.

We decided to look into the lifestyle of young kids, and explore the reasons behind their depression. 

Stage 2: First-hand research: questionnaire round 1

We started to design our first version questionnaire. From our second-hand research, we found out that since kids are staying inside over the quarantine period, they don’t get the chance to play with their friends. We made an assumption that their depression and anxiety could be coming from this loneliness and their desire to stay connected. 

Our questions focus on kids’ interactions with their peers. We collected data of the interaction frequency between their kids and their playdates, of the type toys the parents prefer and their preferred material. We received about 40 responses, but the feedback from these parents were not as satisfying as we imagined. It seemed like they weren’t that worried about their kids not being able to stay connected with their friends. The results showed the same thing: in the question where we asked what are their main concerns for their kids, their concerns were more spread than focused on the “lacking interaction between peers” option. Nevertheless, we summed up the results we acquired and summarized our first version pain point.

Pain Point version 1

Kids cannot connect/play with their playmates during quarantine periods and other times when they are physically departed. 

Similar Toy Design Research and analyze: 

Tinkamo: https://www.tinkamo.com/

Lego Mindstorms (targeting older kids): https://www.lego.com/en-gb/themes/mindstorms

Although focusing on the hardware part,  those all contain digital screen children need to connect with.

Features we designed for Kinnected!:

Narrow down the age rage to 2-5/3-5, for which kids can communicate better.

There should be no screen/non-necessary screen for the toy. Best without a digital screen or app.

Kids need to co-create/compete and win. (design game system)

Consider different kinds of blocks such as sound blocks…

Mother Block (the special one): more complex, embedded Android system, maybe uses Bluetooth/Wifi, reads RFID tags on children blocks. 

Design a game system for the toy. 

Engagement should be virtualizing play-date.

Use wood/natural materials (parents prefer).

Realize audio synchronization.

Considered Problems:

How to keep kids’ attention? (For long-term playing)

What do kids do with blocks when they’re together?

Stage 3: Interview Round 1

Our first interviewee is our professor who has 2 year-old twin daughters. When talking to her daughters, both of them mentioned that they’re really interested in the emoji characters in their mom’s phone. They also love the Peppa Pig cartoon and corresponding toys that allow them to go through the storylines and roleplay with each other. During the conversation with our professor, we asked some open-ended questions so that we can get more comprehensive answers. Since her children are not attending the kindergarten right now, she was worried about their social skills and may not be able to make friends with others. In terms of her children’s interests, she mentioned that she would guide them toward certain fields on purpose by giving them different kinds of educational toys. She also mentioned that her children really enjoy watching cartoons on TV and playing with her phone, as they’re not playing specific games, but only interested in this new kind of interactive experience. 

The second interviewee is a 4 year old girl and her mom. The child likes to listen to stories with Tmall Genie for about two hours a day, and her favorite story is Black Cat. She has a lot of dolls, she likes to talk to the toys and role play with them. “During the pandemic, I couldn’t play with my buddies. I miss them, but it’s nice to play with stuffed animals at home.” Her favorite game is role-playing, no matter what time with whom to play. It works like this: for example, you are a kitten and I am a puppy, and we start making up their own stories. She does not play a lot of video games. Doesn’t have her own social account and doesn’t use her mom and dad’s social account often, but watches videos on Tiktok often. Though having a lot of friends, she likes to stick with her mom all day long. “I don’t like the fact that I can’t go out because of the pandemic, but the rest is fine.”

The mom allows the kid about two hours of play time per day. For the mom having playmates to connect with is important, but not that important. “I’m willing to play with my kids and because I myself am a kindergarten teacher, so I’m with my kid almost all the time. Children love to listen to stories.  I think it’s very educational.” Buy toys ranging from a few to a few hundred yuan, if more than 500 yuan, it needs to be considered twice. Extracurricular activities are: learning ancient Chinese literature and English, and many other children are also learning to draw and dance. It’s very important to keep the child happy.

Stage 4: Second-hand research: questionnaire round 2

However, since the pandemic situation is getting better, the vaccine for the covid has been invented and the government is lifting the stay-at-home order, which allows more outdoor activities for kids. They get to hang out with their playmates more. Then, how to keep them connected when they’re physically departed doesn’t seem like a core problem now. In order to redefine the real pain point, we created a new questionnaire and distributed it to parents of kids under 5 years old. 

Among all of the data we collected, we focused on two key questions that are the most vital to our project. The first one is “what are you most worried about during your child’s absence from kindergarten?” 73.33% of the parents, which is also a majority of them, choose “feeling lonely and unable to play with friends”. “Cannot learn at home” ranks the second, with 46.67% of the parents choosing this option. There is also 42.22% of them choose they’re worried about their kids “Playing electronic products for too long”. Another question is “what are some of the features you expect the toy to achieve?” The top two answers are 1) development of intelligence and early education, with 88.89% of the parents choosing, and 2) let children make more friends / have more communication with friends, with 66.67%. 

Stage 5: Interview Round 2

Our second interview revolved around collecting feedback moreso from the parent as we wanted to adjust our niche and introduce more open ended questions. We decided to interview an NYUSH Business professor named David Hunsaker, who has 6 children ranging from the ages 1-10 years old. Because of the amount of children he had, we thought it would be very insightful for him to tell us what he thought worked in differing age groups, having seen many cycles of kids growing up and fulfilling our target audience. 

During the interview, we asked questions such as:

  • What do you think is your children’s interest for toys
  • Has there been any effect on your kids during quarantine in relation to screen time
  • If you could create a toy for your kids, what do you want feature-wise?
  • What types of toys do you think are the best?
  • As a parent, do you think toys should be educational

From these questions, the conversation opened up to include other insights that the questionnaire was not necessarily considering. Firstly, the professor solidified that socializing for his children (being so many), was not a high priority as compared to the quality of toys and their modularity. When the conversation shifted towards how he would look at toys, he explained that while education was an important factor, more importantly was the ability to captivate his children’s attention span for longer periods of time. He then went on to explain this concept between certain toys having a “one time use” vs “many time use” potential for kids. The main separation had to do with how much the toys could change and provide multi functionality rather than just doing one thing that would over time become very boring. He gave a vivid example of toys he bought that were very modular (such as doll houses that could have many parts to them, or items like cars that could come in a set) vs toys that were overpriced because they would be so 1 dimensional (such as a doll that only says 1 thing, or a plush toy that doesn’t have any types of feedback). He went as far as to say that when him and his wife to the store, it bugged him that there were many toys that would seem hard to distinguish to a first time parent, and that most toys were actually all one time use. In essence, he implied these toys were money eaters with no value add to children, given once the child is bored with the toy, it quickly becomes trash.

Coming out of this interview, we realized the importance of having a “many time use” type of toy and the importance of showing feedback to the children. Without any modularity, our product would flop because it won’t create long standing brand value if it becomes trash quickly. 

Pain Point version 2

Based on our second round research, we have reached some helpful conclusions. First, for parents, what they demand in toys are: 1) educational, 2) healthy, eye-friendly especially, and 3) multi-usability. Then, for kids, they are looking for 1) fun and new experiences, 2) interactive devices, such as smartphones, iPad, or TV, and 3) relate to the popular cartoon characters. However, in the current toy market, parents are struggling to find a screen-less phone/iPad substitution that is fun and educational enough for their kids. Therefore, on the basis of our summary, we develop our new pain point: there is an untapped marketplace where a screen-less toy that has smart capabilities and is educational is highly desired.

Persona → Story

Parents: After re-empathize with our users and re-define the problem, we found out that actually what parents care about most is a kid’s health and education, rather than social connections with peers. For Health, according to our interview, parents of little kids worry more about their kid’s physical health than mental health, which includes eyesights, height, nutrition and etc. among all these, one very pig problem most parents worry about is eye sight. Children are getting exposed to all kinds of digital screens younger and younger, even a two year old child learns how to open and scroll through TikTok. For Nowadays’ young kids’ parents, they struggle to limit children’s time using electronic devices. 

So the second version of the user story is : Grace’s kid often got angry with her because she limited her child’s iPad usage time for one hour a day. She doesn’t want her child to get addicted to ipad games and cartoons because it’s so harmful for the eyes as well as not making her child unhappy. And then, she is introduced with a screen-less and educational device which is fun for children to play. 

Kids: we also redefine and re-empathize their needs: they seek for fun experiences. The desire of playing with playmates is not an essential one. they can have fun themselves if they find the right way to entertain. My cousin, a four year old girl, has fun role playing with friends, her mom, or even just with her dolls. However, They may not need people to play with every time, but they need someone to share the experience with every time. So two key words for kids’ needs are: Fun experience and sharing. For the age ranges, we change it from 2-5 to 2-8 because according to our interview, most parents indicate that they will not purchase a smartphone for their kid before their  kids until the kids get 9 or 10. So we believe, among the range of 2-8, screen-less, educational and fun experience is very important for kids. 

For the user story, candy is now not addicted to electronic devices. One hour of ipad time is enough for her. She has fun with this screen-less device and can learn and share experiences while having fun. 

Prototype

We decided to construct a digital prototype view of how the hardware components would work and how game mechanics could potentially be implemented. Taking inspiration from classical board games (primarily chess and checkers), we wanted to create a board that could easily create dominance in the middle, and would be easy to track movement. It was with these criteria that we decided a 5×5 board would work best for our use case, given it wouldn’t be as complicated as other classic board games, but also not too simple that games would be boring to make for the grid.

With this board, we took the feedback that Wood would be a desirable material t use for construction, and thought about the prototype’s board being made of wood with slotted chamfers to the inside where a lining of LEDs would allow a crevice to fit blocks in. With this, we achieve centering the materials all in 1 orderly fashion, and 1 power source. Also programming would serve to be easier as all the logic would be handled in this board.

Next, we constructed the design for the cubes. We decided that based on our current knowledge of NFC and RFID, it would be best to construct these cube shapes with the idea that each face would be a unique id that could create different game mechanics once in touch with the board. These cubes could serve as the physical element to the board, where movement of these cubes on the board would create different outcomes depending on what the game developer was thinking.

From the construction of this rough prototype, we drafted some ideas on simple games that would provide educational opportunities to children without being too difficult. Games such as matching or using memorization were just some ideas that we would be instructed to construct, as these could be suitable for all age groups. We also constructed some more sophisticated games to showcase how the product has the ability to grow with the children, given that games are super modular and extensive with the interface we have constructed. 

Future Steps

The two main focus “educational” and “fun” for us are still two relatively broad ideas. With the idea of  “no-screen”, we intend to find out small entry points to start our design. Right now, starting from what we’ve got from the interview like designing a more interactive experience with “role play” is one direction we can proceed with. We also consider to start with transforming a digital Ipad game into a physical one. Audio is another element we want to include in the experience. 

Week 14: Final Writing Assignment – Ziying Wang (Jamie)

Dancing With Strangers (Xmas Edition)

link: https://www.youtube.com/watch?v=5r9DHl_vM8Y

Background:

Inspired by the upcoming Christmas, I intended to set the theme for my project as this famous festival. In developing my midterm project: Dancing With A Stranger, my final project continued based on the posenet model, yet turning the previous human-figure based on a more abstract reflection of the body movement. Also inspired by the design of Apple Watch, the sports application in Apple Watch has the strokes of arcs that display different values. I, therefore, implement arc designs in my project to reflect the movement of the users’ hands. To further expand the user group, I enable multiple access to the server which allows more than two users to interact with my project at the same time.

Motivation:

The motivation behind this project is that I want to bring people together during a festival that means to reunion with everyone you know. With devices like computers and mobile phones, we get to celebrate the day with other users on the platform. It’s also considering the ones who are not able to reunion with their family and friends for Christmas in person, with this platform, they get to celebrate this festival in the form of music and dances with the company of others virtually. Additionally, after making Dancing With a Stranger for the midterm project, I was determined to create a platform that allows more than two users so a multiple user platform is a must for my project.

Methodology

For building the multi-player platform, I used node.js to create a communal server. To collect and display all the data tracked by posenet model, I went through the following five steps: store the data, the posenet model takes down the 7 coordinates on the upper half of the user, I stored these data into an array called bodyparts. Draw local data: I draw the figure representing the local user first. Emit data: the code sends the local data to the server which holds the collection of all data. Receive data: the code receives the collection of all data uploaded to the server. Display incoming data: the code then draws out the figures of all incoming data. For creating abstract visual for my project, I used the shoulders as the x-axis and the nose as the y-axis, the center point is the center of the figure. I then took the distance of the nose to the shoulders as the 2/3 of the radius for the outer circle and calculated the angle between the hand and the vertical line as the angle for my arcs’ strokes. For the floating Santa, I stored the previous coordinates of every 50 loops in an array and the Santa would follow the stored coordinates, the gifts bursting out of his sleigh use gravity to stimulate the actual falling. For the color palette, I used the theme colors for Christmas. To create the glowing effect around the ring, I layered many circles with different opacity.

Experiments

The road to get the multi-user platform working is rough. I started with Firebase. I successfully uploaded local data and incoming data to the firebase server but I can’t seem to access and retrieve the data from my code. I, therefore, started using node.js to build the platform. The base code I got for node.js enables multiple users to draw on the same canvas with their mouses. The hard thing to do here is that I need to use node.js to create a communal platform that allows access of all users’ webcams. I used https instead of Http and get certificates for the allowance of accessing webcams. After that, I discover that the display of multiple players’ figures is constantly flickering. I therefore separately draw the figure of different sets of data for the incoming data which helps with the flicker problem. For the visuals, I started with connecting all the coordinates to create a contour of the human, but due to the limitation of the webcam, it is never detecting the bottom half of the human body well, I, therefore, abandoned the bottom half and focused on the abstracting of the upper part. That was when the idea of using arcs came to my mind. The angles made by hands, unfortunately, didn’t lead the movement clearly so I had to multiply the angle by a certain integer. The visual part took me a lot of effort to find the best constant to display the figures.

Social Impact

Dancing With Strangers (Xmas Edition) aims at bringing the world’s lonely people together on a platform and celebrate this festival through dancing, music and the company of one another. The world nowadays is closely connected to our personal devices (laptops, mobile phones, etc.). This project takes advantage of the wide usage of these devices and creates the connection among internet users. Digital devices are often accused as obstacles in strengthening personal connections for people are hiding behind their screens all the time, but I consider them as devices that help build more connections, we get to jump out of our relationship circles and into a more public sphere. Dancing With Strangers allows this to happen by enabling everyone with the link to log onto the platform and interact.

Future Developments

For technical problems, I’ll need to create a more artistic background, currently, the background is just black. I will also need to fix the lagging caused by multiple users log in. I would also add in the coordinates of the legs since only moving with all limbs can be considered as dancing, which would require better webcam detection. Also, I would like to custom the platform for mobile phones users, for now, the platform is typically built for pc users, even though mobile phone users can log onto the webpage with chrome, they can’t enjoy the best experiences. I would also prefer to change my project a little bit, which includes simplify the visual and represent each user with a flickering light, if the user dances while logged onto the webpage, his/hers light would be blinking. The user could see themselves as lights on a world map, and each continent has a different color to represent it. The region with the most flickering light has the most people dancing at the same time.

Week 12 Assignment: Document Final Concept – Ziying Wang (Jamie)

Background

Dancing With Strangers is a development of my midterm project Dancing With a Stranger. In my midterm project, I used Posenet model to mirror human movements on the screen and exchange the controls of their legs. With my final project Dancing With Strangers, I’m hoping to create a communal dancing platform that enables every user to use their own terminal to log onto the platform and mirror all the movements on the same platform. In terms of the figure that will be displayed on the screen, I’m planning to build an abstract figure based on the coordinates provided by Posenet, The figure will illustrate the movements of the human body but will not look like the skeleton or the contour.

Motivation

My motivation for this final project is similar to my midterm project: interaction with electronic devices can pull us closer, but can also drift us apart. Using them to strengthen the connections between people become necessary. Dancing, in every culture, is the best way to pull together different people, a communal dancing platform can achieve this goal. Compared with my midterm project, the stick figure I create was too specific, in a way, being specific means assimilation. Yet people are very different, they move different, Therefore, I don’t want to use the common-sense stick figure to illustrate body movement. Abstract provides us with diversity, without the boundary of the human torso, people can express themselves freer.

Reference

For building the communal dancing platform, I’m using the Firebase server as a data collector that records live data sent by different users from different terminals.

For the inspiration of my abstract figure, I’m deeply inspired by the artist and designer Zach Lieberman. One of his series depicts the human body movement in a very abstract way, it tracks the speed of the movements and the patterns illustrate this change by changing its size. With simple lines, bezier and patterns, he creates various dancing shapes that are aesthetically pleasing. I plan to achieve similar results in my final project.

Some works by Zach Lieberman

Week 11 Assignment: Explore BigGAN or deepDream – Ziying Wang (Jamie)

For this week’s project, I tweaked the parameters in the DeepDream model. 

The tried the examples in the images folder first, then discover that id photo or human portrait works the best with deep dream model. Therefore, I used the picture of Robert Downey Jr. for my deepDream project.

The original picture:

The first set of parameters I changed was the mixed figures.

From left to right, I gradually changed the mixed number from 0 to 0.2, which means for the fourth picture, all four mixed figures in the features become 0.2. The pattern on the picture slowly reduces as the mixed number of each layer increased.

Then I kept the features as they were in the fourth picture, and changed the step size from 0.01 to 0.05 and 0.08 (from left to right). The effect increases as the step size increases, as well as the pattern’s complexity.

The third parameter I tweaked was the number of the octave, I changed it from 3 to 5. The opacity increases while the complexity decreases.

Finally, I increased the iterations from 20 to 50. The opacity of the effect remains the same while the complexity lessens.

After this short tweak with terminal, I uploaded the original picture of rdj to my google cloud and used the deepdream_video_hack to generate the deepDream video.

The video’s frame length depends on the zooming rate and the dreaming rate. Since I didn’t change these rates at first, the video ended quickly. I then changed the dreaming rate to 50 and the zooming rate to 100, then I changed the fps to 150, which should get me a 1 s video, I ended up getting the 1 s fast zooming DeepDream video.

To observe the transformation better, I changed the zooming rate and changing rate to 100, then reduced the fps to 24 frames/s, and generated a 13 seconds long deepDream video that demonstrates the change well.

The zoom_factor of the previous video was too high and the zooming rate was too big, the video was zooming endlessly and the ending became boring. I, therefore, decreased both factors as well as the dreaming rate, so that the time before the zooming can reduce ass well. I ended up getting this video which is quite satisfying.

Train & inference style transfer – Ziying Wang (Jamie)

Model Development:

For this week’s model, I picked a popart image as my style image.

The model training process didn’t start smoothly since the download is constantly breaking down. After that, I uploaded my train.sh file to the DevCloud and started training. I didn’t realize that I already successfully uploaded my train.sh file and summited the training multiple times so I started the training three times, but I ended up getting a trained model after less than 24 hours. 

The result didn’t turn out as well as I had expected:

Pros: I can tell the basic lines and shapes that outline my figure. The shapes of the lines are similar to the ones in the style image.

Cons: The original picture has colors with high contrast, the trained image barely has different colors other than the carnatio, the black and a bit of yellow and blue. Considering these are all the colors from the style image I used, I assume that if my style image consists of more colors, the trained model will contain more colors. 

Video:

Two experiments:

Experiment A:

I tried to connect this new model with the PoseNet model. The GAN model uses the keypressed function to control the transition between the styled image and the camera image. I implemented the PoseNet model in and programmed it to switch between styles when the left and the right wrist positions match. The code works, but not as I imagined since when it switches between styles, the user can’t keep the two coordinates the same. Even when I left some space for the condition (for example, as long as the distance is within 50px it is considered a switch), the transform still glitches all the time.

Experiment B:

I implemented Crystal’s trained GAN model and PoseNet in my model. Trying to make a project that when one person is detected, it shows the style of my GAN model, when two people are detected, shows Crystal’s style, when more than two people are detected, presents the camera footage. Through console.log, I know that my code should work just fine, whereas the actual image is heavily stuck on the webpage. I assume that the three models are too much for my laptop to run.