Wednesday, July 18, 2012

Facial Animation in WebGL

I've recently become interested in 3D facial animation and so I asked the fine folks at what it takes to make a realistic face: link. Naturally the "answer" was many different things. So I decided to focus on just the basics of animating a face. Before I get into my explanation here is link to the final result and an image for those of you without WebGL:

As knight666 explained in the link above, for really realistic facial animation most studios use motion capture, but for lower cost facial animation morph targets are used. A morph target is a snapshot of the current location of all the vertices in a 3D face. Meaning an artist would start out with a neutral face  and then rearrange the vertices to give the face a different emotion. The artist would then save this modified version of the face as a morph target. Facial animation is achieved by blending together these morph targets into a single face.

For example in my demo I have a button that animates the face into a kissing motion, which is achieved by blending together the morph targets for closing the mouth, pucker lips and closing the eyes. So how do you actually blending together several morph targets?

Well if you have 3 morph targets including the base morph target (neutral face) you would do the following for each vertex in the face:
vertexPosition = basePosition + weight1 * (morphPosition1 - basePosition) + weight2 * (morphPosition2 - basePosition)
Essentially we add the difference between each morph target and the base to the base. You'll notice that in the above equation a weight value is being multiplied for each difference of a morph target and base. This weight value is a number between 0 and 1 and it determines how much a morph target affects the final face. This is important because when we animate a face we want to gradually blend in a morph target. No one's face will be neutral one frame and mad the next, the face will gradually become mad over the course of a few seconds.

Unfortunately there is a limit to the number of morph targets WebGL can blend together at the same time, because WebGL shaders have a maximum number of vertex attributes that can be bound at once. And each morph target will bind to at least one vertex attribute. In my demo's case for each morph target I would bind the morph target's vertex positions to one attribute and its vertex normals to another. So I use up 2 attributes for each morph target.

According the data on it seems that nearly all WebGL browsers support up to 16 vertex attributes (and very few support more).

Because of this limitation I chose to make my demo blend together at most 5 morph targets (or technically 6 if you count the base morph target).

But because a 3D face will normally have far more than 5 morph targets to choose from, you need to dynamically bind the morph targets you are currently using to the graphics shader. I did this by binding the first 5 morph targets with non-zero weights that I could find.

It is important for me to mention that the face I'm using in this demo was obtained from this blender file created by Taha Algherbawy. In addition the concept of morph targets is actually known as shape keys in blender, so my code uses the words "shape key" rather than "morph target".

Lastly in order to get javascript readable data files of the 3D face, I wrote a python script that extracts all the shape key related information out of blender into JSON format. This script and the rest of the code for the demo can be found here.