Hundreds of AI Apps with Hidden Deepfake Features Found on App Store and Google Play

Discovery of Hidden Deepfake Functionality

An investigation uncovered a network of more than 400 AI applications available on both Apple's App Store and Google Play Store. These apps, attributed to a single Chinese organization, contained concealed features that allowed users to generate sexual and violent deepfakes from uploaded photos. The deepfake scenarios included explicit and threatening imagery.

Deceptive App Review Tactics

The apps bypassed app review processes by initially presenting as benign services, such as language tutors or workout coaches. Their hidden deepfake capabilities were activated under specific conditions, such as after a server check or when downloaded via ads promoting the explicit functionality on platforms like Facebook and Instagram. Once activated, users could access pornography templates and purchase credits to create deepfakes.

Platform Responses

Apple removed 19 of the identified apps after receiving details of their hidden features. Apple stated that the apps initially complied with their rules but developers later changed the content served. Meta, which hosted thousands of ads for these apps, took action against over 10,000 ad accounts linked to the domains associated with these applications. Google also had these apps on its Play Store.

Implications for Content Moderation

This incident highlights challenges in app store content moderation, particularly with developers employing bait-and-switch tactics to circumvent policies against harmful content. Despite clear policies from Apple, Google, and Meta prohibiting such deepfake material, these apps operated for up to a year, indicating a gap in detection mechanisms for dynamically loaded or conditionally activated content.