Follow ZDNET: Add us as a preferred source on Google.
ZDNET’s key takeaways
- Anthropic, Google DeepMind, and OpenAI scored the highest.
- Even those three only narrowly received passing grades, though.
- The “existential safety” category received especially low scores.
The world’s top AI labs aren’t exactly scoring top marks in their efforts to prevent the worst possible outcomes of the technology, a new study has found.
Conducted by nonprofit organization Future of Life Institute (FLI), the study gathered a group of eight prominent AI experts to assess the safety policies of the same number of tech developers: Google DeepMind, Anthropic, OpenAI, Meta, xAI, DeepSeek, Z.ai, and Alibaba Cloud.
Also: Is DeepSeek’s new model the latest blow to proprietary AI?
Each company was assigned a letter grade according to six criteria, including “current harms” and “governance & accountability.” The assessments were based on publicly available materials like policy documents and industry reports, as well as a survey completed by all but three of the eight companies.
Anthropic, Google DeepMind, and OpenAI scored the highest, but even they received what in an academic setting would barely be considered a passing grade: C+, C+, and C, respectively. The other five scored even lower: all Ds, except for Alibaba Cloud, which received the lowest grade of D-.
Credit: Future of Life Institute
“Even the strongest performers lack the concrete safeguards, independent oversight, and credible long-term risk-management strategies that such powerful systems demand, while the rest of the industry remains far behind on basic transparency and governance obligations,” the FLI wrote in a report summarizing its findings. “This widening gap between capability and safety leaves the sector structurally unprepared for the risks it is actively creating.”
Also: I tested DeepSeek’s R1 and V3 coding skills – and we’re not all doomed (yet)
It’s a bleak performance review for some of the industry’s most widely used and powerful AI models. The results could also add pressure to other companies to develop and implement effective safety measures, at a time when competition among tech specialists is escalating rapidly.
Existential risk?
The most concerning finding from the new study is the group-wide dismal scoring in the category of “existential safety,” which the FLI defines as “companies’ preparedness for managing extreme risks from future AI systems that could match or exceed human capabilities, including stated strategies and research for alignment and control.”
Credit: Future of Life Institute
The question of whether or not AI could ever pose a threat to humanity’s survival, on par with a global pandemic or a nuclear armageddon, is very much up for debate. So-called AI “boomers” tend to dismiss such fears as alarmist while arguing that the social and economic benefits of AI eclipse potential downsides. Their “doomer” counterparts, meanwhile, tend to warn that the technology could escape human control and potentially destroy us in ways that are difficult to predict.
The debate about the impact of AI has been intensified by the tech industry’s recent embrace of “superintelligence” as a marketing buzzword and technical goalpost. You can think of this trend as artificial general intelligence (AGI) — an AI system that can match the human brain on any cognitive task — on steroids: a computer so mind-bogglingly more advanced than our own brains that it would exist on an entirely different, categorically higher level of intelligence, like the difference between your own intellect and that of a nematode.
Companies like Meta and Microsoft have explicitly stated their ambitions to be the first to build superintelligence. However, it isn’t at all clear what that tech might look like when instantiated in consumer-facing products. The goal of the FLI study was to call attention to the fact that companies are racing to build superintelligence in the absence of effective safety protocols to keep such advanced systems from spinning out of control.
Also: Mistral’s latest open-source release bets on smaller models over large ones – here’s why
“I believe that the best disinfectant is sunshine, that by really shedding light on what companies are doing we give them an incentive to do better, we give governments an incentive to regulate them better, and we just really increase the chances that we’re going to have a good future with AI,” FLI president and MIT physicist Max Tegmark said in a YouTube video summarizing the findings of the new study.
The nonprofit also published a statement in September, which was signed by AI “godfathers” Geoffrey Hinton and Yoshua Bengio, among other prominent tech figures, calling for an industry-wide pause on the development of superintelligence until industry leaders and policymakers can chart a safe path forward.
Moving forward
In broad strokes, FLI’s message to each of the eight companies included in the study is the same: it’s time to move beyond just paying lip service to the need for effective AI guardrails and to “produce concrete, evidence-based safeguards” to prevent the worst-case scenarios.
The research also offered specific recommendations to each of the eight companies based on their individual grades. For example, this was the advice given to Anthropic, which scored the highest across all six categories: “Make thresholds and safeguards more concrete and measurable by replacing qualitative, loosely defined criteria with quantitative risk-tied thresholds, and by providing clearer evidence and documentation that deployment and security safeguards can meaningfully mitigate the risks they target.”
But the recommendations are just that, and no more. In the absence of comprehensive federal oversight, it’s difficult, and perhaps impossible, to hold all tech companies accountable to the same safety standards.
Regulatory guardrails of the kind that currently exist around industries like healthcare and air travel are in place to make sure manufacturers create products that are safe for human use. For example, drug developers must complete a multiphase clinical trial process, as mandated by the Food and Drug Administration, before a new pharmaceutical product can be legally sold on the market.
Also: What is sparsity? DeepSeek AI’s secret, revealed by Apple researchers
There’s no such federal body to oversee the development of AI. The tech industry is more of a Wild West, with the onus for customer protection (or not) falling predominantly on the companies themselves, though some states have implemented their own regulations.
However, public awareness about the negative impacts of AI, at both a societal and individual level, is growing: both OpenAI and Google are currently embroiled in lawsuits alleging that their AI systems have led to suicides, and Anthropic’s Claude was reportedly used in September to automate a cyberattack on behalf of state-supported Chinese hackers.
The upshot of this negativity is that, even in the absence of robust federal oversight, reckless development of AI tools — releasing new iterations of chatbots without equally sophisticated safety mechanisms — could become so taboo in the AI industry that developers become incentivized to take it seriously.
For now, though, speed over safety still seems to be the guiding logic of the hour.
Takeaway for users
The lack of federal regulation of the AI industry, coupled with the race between tech developers to build more powerful systems, also means that users should educate themselves about how this technology can negatively impact them.
Some early evidence suggests that prolonged use of AI chatbots can distort one’s worldview, dull critical-thinking skills, and take other psychological tolls. Meanwhile, the proliferation of AI tools and their integration into existing systems that millions of people already use make the technology increasingly difficult to avoid.
While the FLI study probably won’t suddenly cause a widespread change in tech developers’ approach to AI safety, it does offer a window into which companies are offering the safest tools, and how those tools compare with one another along particular domains.
For anyone interested not only in the potential existential harms of AI but also in the risks they present to individual users, we recommend reading Appendix A in the full report to get a fine-grained perspective on how each of the eight companies performed across specific safety measures.