Inside the marketplace powering bespoke AI deepfakes of real women

Civitai, an online marketplace for AI-generated content backed by venture capital firm Andreessen Horowitz, has been found allowing users to purchase custom instruction files for creating celebrity deepfakes, with some specifically designed to generate pornographic images banned by the platform, according to a recent analysis by researchers at Stanford and Indiana universities. The marketplace, which operates as a hub for sharing AI models and custom instructions known as 'LoRAs,' has become increasingly popular for generating highly realistic images of real people, including celebrities. Despite claiming to prohibit non-consensual explicit content, the researchers discovered that numerous instruction files were being sold with the express purpose of bypassing these restrictions. These files contain specific parameters designed to generate pornographic content featuring real individuals without their consent, raising serious ethical and legal concerns. The study highlights the growing challenge of regulating AI-generated content as technology becomes more sophisticated and accessible. While platforms like Civitai attempt to set guidelines against harmful content, the decentralized nature of AI model development and sharing makes enforcement particularly difficult. The involvement of a prominent venture capital firm like Andreessen Horowitz adds another dimension to the issue, suggesting that mainstream investment may be accelerating the problem without adequate safeguards.