Why has Google not released its generative text-to-image AI model, Imagen Video?

Google has withheld the release of Imagen Video due to concerns about its potential misuse for generating fake, hateful, explicit, or harmful content.

What challenges do Google researchers face in filtering problematic content?

While much explicit and violent content can be filtered, Google researchers find social biases and stereotypes challenging to detect and filter.

Google Withholds Imagen Video Release Over Explicit Content Concerns

Q: What kind of content does Google categorize as problematic for its AI models?

Google researchers categorize explicit sexual material alongside fake, hateful, or harmful content, and depictions of human sexuality as problematic data.

Google delays Imagen Video's release due to its AI generating explicit, hateful, and harmful content, categorizing it with problematic data. The company aims to mitigate misuse concerns before launch.

Google researchers continue to categorize explicit sexual material alongside "fake, hateful, or harmful content" requiring filtering, according to a new paper on the company's proprietary AI technology. This stance has led Google to withhold the release of its generative text-to-image AI model, Imagen Video, until concerns regarding its potential misuse for generating such content are addressed.

AI and Content Filtering

In a research paper, Google scientists noted that generative text-to-image AI models, such as Dall-E 2, have demonstrated "tremendous progress." However, Google has opted not to release its own version, Imagen Video, until "concerns are mitigated" regarding its potential misuse. These concerns include the generation of "fake, hateful, explicit or harmful content."

Tweak Town, a tech news site that initially highlighted the Google paper, editorialized that Google "has subtly said that it won't be releasing its new video-generating artificial intelligence system over it producing gore, porn and racism." Google's researchers and policymakers consider depictions of human sexuality as part of "problematic data" that presents "important safety and ethical challenges."

The researchers expressed optimism about developing improved tools for censoring sexual content in the future. However, they concluded that "though our internal testing suggests much of explicit and violent content can be filtered out, there still exists social biases and stereotypes which are challenging to detect and filter." Consequently, Imagen Video will not be released until the company can fully censor "problematic content," including "explicit material."

A paper cited by the researchers directly addressing explicit content is titled "Multimodal Datasets: Misogyny, Pornography, and Malignant Stereotypes." This paper categorizes "pornography" with "troublesome" material.

Google's Broader Censorship Practices

Google and its subsidiary companies, including YouTube, have a history of removing or omitting information from their services. These actions are taken to comply with company policies, legal demands, and government censorship laws. Numerous governments have requested Google to censor content. In 2012, Google approved more than half of the requests it received through court orders and phone calls. This figure did not include China or Iran, which had completely blocked Google or one of its subsidiary companies.

As of 2025, Google continues to receive hundreds of thousands of removal requests annually from governments globally. These requests are most commonly related to national security, copyright, or defamation. Google regularly reviews each request for compliance with its policies.

AdSense Policies and Content Restrictions

Google's content policies extend to its advertising platform, Google AdSense. In February 2003, Google ceased displaying advertisements from Oceana, a non-profit organization protesting a major cruise ship operation's sewage treatment practices. Google cited its editorial policy, stating that "Google does not accept advertising if the ad or site advocates against other individuals, groups, or organizations."

In April 2008, Google declined to run ads for a UK Christian group that opposed abortion. The company explained that "At this time, Google policy does not permit the advertisement of websites that contain 'abortion and religion'." In April 2014, Google removed ads for certain crisis pregnancy centers following an investigation by NARAL.

Key Facts

Google researchers classify explicit sexual material with "fake, hateful, or harmful content."
Google has withheld the release of its AI model, Imagen Video, due to concerns about its potential to generate "explicit or harmful content."
A cited paper, "Multimodal Datasets: Misogyny, Pornography, and Malignant Stereotypes," groups "pornography" with "troublesome" material.
Google has a history of removing content and ads to comply with company policies, legal demands, and government censorship laws.
As of 2025, Google receives hundreds of thousands of annual removal requests from governments worldwide.