2021 DURF: Synchronizing 3D Animation with Piano Performance
Synchronizing 3D Animation with Piano Performance
2021 Deans’ Undergraduate Research Fund (DURF)
Demo
Over All Accomplishments
In this project, I successfully built up the connection between MIDI signals (received from virtual/physical MIDI keyboard) and 3D character animations utilizing the three.js and p5.js libraries. During the technical development, I made and remade 3D models and tried various export settings and plug-ins. I also tested different loaders in three.js in order to load the exported model successfully. (Here’s a screenshot of my working files. For detailed process explanations please refer to my DURF Process Documentation):
After multiple testings and trials, I finally located the problems of three.js model&animation loading. I self-learned another 3D software Blender and combined its advantages in animation exporting with the proficient modeling UI in Maya. Under the professional guidance of Professor Moon, I successfully incorporated the Web MIDI API into my three.js project so that I can receive and utilize the MIDI signals (from either a virtual keyboard or an electronic piano) to trigger the character’s movement in real-time. I managed to develop an efficient workflow:
Through the execution of this project, I prove, technically, the feasibility of visualizing music through 3D character animations online.
Explanation of Changes
Originally, I wanted to generate and render 3D character animations that synchronize my performance of a classical piano solo “Ballade No.2” in real-time. By making the 3D characters and camera moving according to the received MIDI signals, I wanted to narrate a story between Eggy and Cube (written by myself) in accord with the speed of my piano performance. During the execution, I found three major problems: 1) The connection between the overall story and the classical piano piece is awkward; 2) In some of the segments, the intervals between two chords are very short, making it hard to catch every signal during the live performance; 3) Due to the unexpected frustrations in three.js model loading, it was almost impossible to complete the project with the original scope.
In early July, I made my mind to reconstruct the whole project. I redesigned and remodeled a 3D character “Brushman” with limbs and hairs being brushes:
He will dance to a simple melody (one-note-based song) without a given narration/story. Each of his movements is assigned a single piano note instead of complicated chords (e.g. C7-jump, B6-idle, etc.). While dancing, the Brushman’s brushes that contacted the floor will leave traces on the floor. The trace’s color will vary according to the note:
In this way, the connection between music and the 3D visualization is strengthened and the scope of the project is reduced. I can now focus on solving the technical problems and at the same time ensure the overall quality and artistry of this project.
Lessons Learned
Through researching and conducting the project, I not only strengthened my skills in 3D animation production (modeling, UV mapping, rigging, animating, new software using, etc.) and coding skills (Three.js, Web MIDI API, etc.), but also improved my research ability.
Throughout this summer, I’ve been testing out different settings and workflows that support the 3D animation loading in three.js. In the process, I modeled four versions of 3D characters in Maya:
By testing out different modeling methods (topology/sculpting in Mudbox), I get a proficient
understanding of the modeling practices as well as the pros & cons of different modeling methods. What’s more, I tested out different rigging methods both in Maya and in Blender. In Maya, I learned to rig using 1) traditional rigging ways; 2) Advanced Skeleton plug-in; and 3) Maya default Quick Rig. Also, I learned to use Blender Rigify plug-in when all the previous rigging methods failed. Through practicing different rigging methods, I understood better the principles and mechanisms behind rigging, which can be applied to all kinds of rigging methods.
V4:
Besides the modeling and rigging, I also learned to use Blender NLA tracks, which greatly promotes the efficiency of animation making. In contrast to the 3D animation production, both three.js and Web MIDI API are new fields for me. Under the professional guidance of Professor Moon, I learned to set up the project, load models using different loaders and multiple built-in functions and methods in three.js.
In addition to the technical skills, I learned how to conduct a research efficiently. At the beginning, I tried to give considerations to all three areas (modeling and animating, three.js set up and learning Web MIDI API) at the same time. I was overwhelmed by the new knowledges and became inefficient. Professor Moon corrected my method in time. He suggested me focus on solving the loading problems in three.js first and only start exploring MIDI and 3D animations after the loading problems were solved. I successfully lifted my research efficiency in this way. During the research, I also develop my workflow:
Overall, I gained a concrete understanding of the technical skills that support my music visualization project as well as my research ability, which will lay the foundation for my future project development.