Korea moves fast on Grok as global deepfake backlash grows

Screens display the logo of Grok/ AFP–Yonhap
SEOUL, January 15 (AJP) — South Korea has taken its first regulatory steps targeting xAI’s chatbot Grok, as governments worldwide step up scrutiny of generative AI tools over deepfake abuse and illegal content.

The Korea Media and Communications Commission (KMCC) on Wednesday sent an official notice to X requesting stronger safety measures for Grok, including tighter controls on harmful content and stricter restrictions on minors’ access. 

The commission also demanded clarification on Grok’s training data sources, content-filtering standards and accountability structures for services provided to South Korean users.

According to Shin Yoon-jae, a deputy director at the KMCC, the notice emphasized that any platform operating in South Korea must comply with the Information and Communications Network Act, including its legal obligation to protect minors.

“We have asked X to put safeguards in place and to report what measures it is taking,” Shin said. “At the same time, we are preparing legislation to strengthen safety standards for generative AI. Until then, we are tightening regulatory monitoring and have requested concrete plans to prevent the spread of AI-generated images and videos.”

Seoul’s move reflects growing concern that generative AI tools are lowering the barrier to producing illegal and abusive content, particularly non-consensual deepfake imagery.

Regulators say conversational AI systems like Grok make it possible to generate realistic images or scripts using simple text prompts, expanding misuse beyond technically skilled actors. This ease of access has fueled the rapid spread of fabricated sexual images, impersonation scams and manipulated visuals, often outpacing platform takedown mechanisms.

One of the most pressing risks involves non-consensual sexual deepfakes, including content involving minors. Digital safety groups and law enforcement agencies report that such material can be produced and circulated within minutes, causing lasting psychological harm, reputational damage and, in some cases, extortion of victims.

Concerns have also grown over opaque moderation systems. Authorities argue that limited disclosure about training data, filtering thresholds and internal accountability makes it difficult to assign responsibility when harmful outputs occur — a potential violation in jurisdictions with strict child-protection and privacy laws.

The backlash against Grok has triggered regulatory action beyond Korea.

In the United Kingdom, online safety regulator Ofcom has opened a formal investigation into X under the Online Safety Act, following reports that Grok was used to create and share non-consensual sexual images and content involving children. The probe will assess X’s compliance with duties related to risk assessment, illegal content prevention, privacy protection and child safety.

In Southeast Asia, regulators in Indonesia and Malaysia temporarily blocked access to X over the weekend, citing concerns over the misuse of generative AI to produce sexually explicit and non-consensual content.

While the United States is unlikely to intervene directly in the near term, experts say international pressure could force platforms to tighten safeguards.

“Countries with stronger online safety laws can shape global standards,” said Mary Anne Franks, a law professor at George Washington University. “When enforcement actions accumulate, platforms often respond by raising protections across markets.”

In South Korea, officials say the urgency is practical rather than theoretical. Reports of AI-generated explicit content, impersonation scams and manipulated images affecting domestic users have risen, even when the platforms involved are based overseas.

Regulators argue that reactive takedowns are no longer sufficient in an environment where AI-generated content can go viral before authorities or platforms become aware of it.
Ryu Yuna Reporter Julia37@ajupress.com