This week I tried out the AI video style transfer pipeline. Initially I plan to train a Lora model of myself and put myself into the scenes of “How to train your dragon”, but a series of mishaps happened and I didn’t get a working model before Friday. So for now I am doing the style transfer pipeline first, and I’ll try the face switching pipeline over the weekend.
I used Winsey’s arcane art model for this project, so much thanks to Winsey!
The pipeline itself seems simple: train a model of the style I want to apply to the video, use Comfy to do style transfer of the key frames which happens when there’s a perspective change, use EBsynth to sync those key frames to the video, and viola! It should be done.
The real process takes a lot more time than I thought it would. The first step, training the model is itself quite time consuming (perhaps due to the fact that everyone’s trying to use the public computer in the last days lol), then generating a satisfying image is very tricky. Sometimes the image looks nice but does not align with the orginal image, sometimes there are extra features added, and all times the key frames don’t correspond with one another. Here are some of the weird looking failed attempts:
And these are what I ended up using:
I photoshoped them to make them look better, for example the smallest boat is impossible to get right by AI, so I just PSed my original frame’s boat into the generated picture. I also painted their clothes so they look roughly the same.
And this is what I’ve got in the end:
https://www.dropbox.com/scl/fi/5qdc4dyxm214vqtd3vyjc/BoatInFin.mp4?rlkey=ebbtadt1spjxfq3kerw1ry8s8&st=zpt8o9uj&dl=0
It’s looking… okay, I guess? I feel that adding another frame in the middle perhaps will improve things a bit, but I couldn’t get a generated picture that alignes with the other frames, so it ended up looking more confusing. It is pretty cool though to see it moving. I added the breakdown of how this scene was built from Unreal Engine. I think if there’s a way to ensure a better alignment for the generated images, this pipeline could work really well.
As I’m typing my other model just got ready! I’m excited to try it out soon:)