Zao, a new mobile app coming out of China, can replace characters in TV or movie clips with the user’s own facial picture within seconds, raising new privacy and fraud concerns.

Developed by Momo, the company behind Tantan, China’s answer to Tinder, Zao went viral shortly after it was made available on the iOS App Store in China, Japan, India, Korea, and a couple of other Asian markets. It allows users to swap a character in a video clip for the user’s own face. The user would choose a character in a clip from the selections, often iconic Hollywood movies or popular TV programs, upload his or her own picture to be used, and let the app do the swapping in the cloud. In about eight seconds the swap is done, and the user can share the altered clip on social media.

While many are enjoying the quirkiness of the app, others have raised concerns. First there is the concern for privacy. Before a user can upload their pictures to have the app do the swapping, they have to log in with their phone number and email address, literally losing face and giving away identification to the app. More worryingly, the app, in its earlier version of terms and conditions would assume the full rights to the altered videos, therefore the rights to the users’ images.

Another concern is fraud. Facial recognition is used extensively in China, in benign and not so benign circumstances alike. In this case, when an altered video with the user’s face in it is shared on social networks, it is out of the user’s control and will be open to abuse by belligerent parties. One of such possible abuses will be payment. Alipay, the online and mobile payment system of Alibaba, has enabled retail check-out with face, that is, the customer only needs to look at the camera when leaving the retailer, and the bill will be placed on the users’ Alipay account. By adding a bit fun into the process, check-out by face not only facilitates retail transactions but also continuously enriches Alibaba’s database. (It would not be a complete surprise if this should be one reason behind the euphoria towards AI voice by Jack Ma, Alibaba’s founder.) The payment platform rushed to reassure its users that the system will not be tricked by the images on Zao, without sharing details on how.

Though Zao is not the first AI-powered deepfake application, it is one of the best worked out, therefore most unsettling ones. In another recent case, involving voice simulation and the controversial scholar Jordan Peterson, an AI-powered voice simulator enabled users to type out sentences up to 280 characters for the tool to read out loud in the distinct, uncannily accurate Jordan Peterson voice. This led Peterson to call for a wide-ranging legislation to protect the “sanctity of your voice, and your image.” He called the stealing of other people’s voice a “genuinely criminal act, regardless (perhaps) of intent.”

One can only imagine the impact of seamless image doctoring coupled with flawless voice simulation on all aspects of life, not the least on the already abated trust in news.

The good news is that the Zao developer is responding to users’ concerns. The app said on its official Weibo account (China’s answer to Twitter) that they understood the concerns about privacy and are thinking about how to fix the issues, but “please give us a little time”. The app’s T&C has been updated following the outcry. Now the app would only use the uploaded data for app improvement purposes. Once the user deletes the video from the app, it will also be deleted in the cloud.