Face2Face: Scientists invent futuristic face capture that makes celebrities say anything you like

Computer scientists from Germany and the US have invented a new futuristic face capture technology that can map a user's facial expressions in real-time onto a famous celebrity and then generate realistic video showing the celebrity saying anything the user chooses.

Researchers from the University of Erlangen-Nuremberg, the Max Planck Institute for Informatics and Stanford University have developed a system that is able to track facial expressions from someone using an RGB web cam and the facial expressions of say Donald Trump, Barack Obama or George W Bush from YouTube video interviews using a dense photometric consistency measure.

The Face2Face system then deforms the user and the face of the celebrity and looks for the mouth interior that best matches the re-targeted expression and warps it to fit onto the celebrity's face.

Once the user's expression is rendered on top of the celebrity's video interview sequence, the video comes out looking realistically as if the celebrity is actually mouthing those words or making the same facial expressions as the user.

"Our method introduces a new RGB tracking pipeline which we compare against state-of-the-art face-tracking methods," the researchers state In their video.

The results are hilarious and downright uncanny, and although there's no way to replicate the celebrity's speech to say what the user wants them to say (that will be up to someone else to research), this means that in future the technology could in theory be used to fake entire interviews with famous people.