As AI-enabled digital gender violence proliferates quickly, critics say the current response system—centered on takedown requests—is failing to stem the spread of harm. They are calling for proactive prevention measures, including integrating gender-equity perspectives into technology design and strengthening platform accountability before harm occurs.
A response system built around after-the-fact takedowns
At a seminar on April 16 at the Korea Women’s Development Institute in Eunpyeong District, Seoul, policymakers and researchers discussed concrete steps to confront digital gender violence amid expanding AI use. As commercial AI becomes widespread, the production and distribution of sexually exploitative material are growing more organized and automated. In an environment where a single text prompt can generate explicit content in seconds, market mechanisms that turn victims’ abuse into profit are expanding. Many victims remain unaware that abuse has occurred, and when deepfake bots and anonymous Telegram channels separate creators from distributors and enable rapid deletion of evidence, it becomes increasingly difficult to identify perpetrators.
Speakers argued that the prevailing focus on post hoc takedowns cannot keep pace with the speed at which abuse spreads. For example, about 300,000 video removals were supported in 2024 for digital sex crimes, but many offending sites were hosted on overseas servers, limiting the effectiveness of takedowns; content was often copied and reposted rapidly before removals took effect. In a media ecosystem where distribution volume directly drives revenue, expecting platforms to self-regulate is often unrealistic.
Kim Aera, a research fellow at the Korea Women’s Development Institute, said, “After-the-fact responses are necessary, but we must find proactive approaches. If gendered violence leverages technological infrastructures, interventions must create technical friction at the stages where abuse is generated and distributed.” In other words, she argued, raising the cost and difficulty of creation and distribution can reduce both the frequency and scale of abuse.
Kim urged development of gender-aware AI. She said AI systems should be designed to systematically assess and block gendered power imbalances, intersecting inequalities, and structures of violence across the lifecycle. To that end, she proposed making mandatory the review of gender bias and gender-based violence risks during the technology design phase. She also recommended establishing Korean-context prevention standards for gendered violence to be embedded in chatbots, educational AI, and customer-service AI systems.
Speakers also criticized the Basic AI Act, enacted in January, for failing to address risks related to gendered violence. Kim argued that the act’s "high-impact AI" category—targeting systems that could seriously affect life, physical safety, or fundamental rights—should explicitly include gender-violence risks. She added that government agencies responsible for rapid responses to digital gender violence are fragmented across ministries and called for a more efficient cross-ministerial cooperation framework.
“If an interagency cooperation body exists, we should establish obligations for cooperation and public reporting,” Kim said. “We also need to define what constitutes gendered violence, develop a 'gender safety index' to measure those risks, and subject that process to public deliberation.” She emphasized the need to look beyond after-the-fact measures and prioritize design-stage prevention, calling for an active effort to extend gender-equity policy—from laws, institutions, and practices—into the AI technology domain.
Strengthen platform accountability to prevent harm
Preventing digital gender violence also requires clearer platform roles and responsibilities. In the second presentation, Jung Yeonju, deputy research fellow at the Korea Women’s Development Institute, said, “The state’s ability to control the distribution of child and youth sexual exploitation online is limited without platform cooperation. Platforms are required to report and remove illegal content when detected, but relying solely on takedowns cannot prevent proliferation. We need stronger platform accountability.”
Other countries have adopted different regulatory models. Indonesia’s MR5 requires private electronic system operators to register with national systems and empowers the state to sanction platforms that host prohibited content. If the government orders removal and a platform fails to comply, authorities can issue written warnings, fines, or block access. In January, Indonesia blocked the AI chatbot 'Grok' for producing and distributing child sexual exploitation material; regulators allowed a conditional resumption only after the operator pledged measures to prevent recurrence.
The European Union takes a risk-based, differentiated approach rather than imposing identical obligations on all platforms. Very large online platforms—those with at least 45 million monthly EU users—must regularly assess risks tied to service design, algorithms, content moderation, advertising systems, and data practices. They are required to proactively evaluate and manage structural risks such as harms to minors, the spread of illegal content, human-rights infringements, and gender-based violence. Violations can trigger fines up to 6% of global annual turnover.
The UK’s Online Safety Act imposes legal duties on major platforms that significantly affect minors, requiring safety to be embedded across service design and operations. Ofcom sets specific standards for risk assessments and protective measures and oversees compliance. In Australia, the communications minister establishes online safety expectations, and the eSafety Commissioner monitors compliance, issues guidelines, and oversees reporting and takedown enforcement. Those expectations include reporting systems, harm minimization, and preventive measures to block children’s access to particular harmful materials.
Jung said, “When Korea designs preventive policy, it should examine overseas legislative examples. Policymakers must decide whether to design regulation that focuses on outcomes or to build sustainable institutional structures, and they must balance platform autonomy with appropriate state intervention.” She added, “Regulatory frameworks must strike a balance between clarity and flexibility, and the regulator’s role and capacity must be carefully designed. It is important to determine whether the government will set standards while a dedicated agency enforces and supervises them. Effective prevention requires active public-private commitment and shared responsibility.”
