Google Gemini AI hacked 3 companies by itself during a security test

Gemini reportedly found “public information online and guessed credentials to access websites it thought were part of the test,” a Google official told the BBC, adding that in each instance, “the model stopped.”

According to reports, the three affected companies have been informed about the incidents.

The development comes amid growing debate among technology experts over whether the pace of AI development should be slowed because of concerns about the potential risks posed by increasingly capable AI systems.

Heather Adkins, vice president of Security Engineering at Google, told the BBC: “We ensured the three entities were made aware, and we worked with our training partner on the changes they’ve now made to their testing processes.”

“These events highlight the importance of training powerful AI models to act responsibly,” she added.

The incident also comes as the United Nations continues to warn about the rapid advancement of artificial intelligence and the need for international cooperation to manage its potential risks.

The UN has said AI is advancing rapidly, raising questions about how societies and governments can capture its benefits while addressing risks that often extend beyond national borders.

Concerns have intensified in recent weeks following warnings from AI researchers and experts about the potential long-term risks posed by increasingly powerful systems.

For the UN, one of the key ways to address the global risks associated with AI is through international cooperation, including shared principles and commitments on AI governance.

Coordination is ‘indispensable’

“National action is essential, but global coordination is indispensable,” UN Secretary-General António Guterres said on Wednesday.

The UN has worked for several years to provide a forum for countries to develop a shared understanding of AI technologies and discuss cooperation and regulatory frameworks covering issues such as safety, human rights and disparities in access.

Rules-based development

The UN’s current work on AI governance builds on the Global Digital Compact, which was adopted by member states in 2024 as part of the Pact for the Future.

The framework seeks to promote international cooperation on digital technologies, including artificial intelligence, while addressing associated risks and ensuring that technological development remains inclusive and guided by shared principles.