Digital Clones and Virtual Celebrities
Facial Tracking, Gesture Recognition and Animation for the Movie Industry
Digital Cloning System™ is a high performance facial tracking and animation
solution specifically designed to meet the needs of the entertainment industry.
Special and digital effects are now being used in most feature films, thus representing
an important new application domain for facial and body tracking technologies. Virtual
Celebrity’s system consists of a real-time module responsible for pre-visualization
of the tracked data on the face of an animated celebrity, and an off-line component
that suitably meets the needs of high quality special effects and film production.
Automated analysis and synthesis of facial expressions have been in the focus of
face recognition research for the last decade. Experimental systems have been developed
and tested for model-based coding techniques that achieve visual communication at
very low data rates and high compression of 3D animated data. More recently, the
advent of Internet-driven avatar technology has created a strong need among users
to assume a particular identity in virtual spaces and to freely interact with each
other. For both applications, the principal front end is a real-time facial motion
tracker that follows the motion of a small set of characteristic points on the face
(typically less then 20), and uses gesture recognition to analyze, transmit, and
re-animate events related to expressions and emotions.
Virtual Celebrity Productions has designed and built a proprietary Digital Cloning
System™ (DCS) which utilizes a dual tracker architecture and advanced facial
animation tools to create photorealistic ‘seamless’ human characters.
The two major components of our tracker module are:
- A real-time tracking & animation system called Geppetto
- A very high quality off-line tracking/rendering pipeline. The DCS module accepts
a sequence of film resolution images (4K by 4K pixels, 16 bit / color) and outputs
tracking data and high quality 3D animation ready for film production. The specific
system incorporates:
- A real-time facial tracker and gesture recognizer coupled with animation software
used to pre-visualize the effects of current tracking data.
- A set of specialized facial feature trackers that extract all the necessary information
relevant for creating a photo-real Digital Clone™.
- A camera tracker that derives the 3D motion of the camera as needed for special
effects.
- A highly accurate off-line facial feature tracker that is capable of tracking up
to 500 points without markers on an actor’s face.
- A deformable model-based 3D face tracker based on the integration of optical flow
and edges extracted from the given image sequence.
- Interfaces to the major computer animation packages allowing for direct control
of any muscle or surface deformation-based facial animation model.
