#Safe Superintelligence

[themorningnewsinformer]

OpenAI Uses Google’s AI Chips to Power ChatGPT Shift

Introduction: OpenAI Rethinks Infrastructure with Google AI Chips OpenAI AI chips, the company behind ChatGPT, is undergoing a significant infrastructure shift by using Google’s AI chips—specifically Tensor Processing Units (TPUs)—to power its AI models. Previously dependent on Nvidia GPUs and Microsoft’s Azure cloud, this move signals OpenAI’s growing need for diversified and scalable compute…

[muridigital]

Former OpenAI chief scientist set to start new AI company

ChatGPT maker OpenAI’s co-founder and former chief scientist Ilya Sutskever is starting a new artificial intelligence company. The company is focused on creating a safe AI environment at a time when some of the biggest tech companies are looking to dominate the generative AI boom. The company is called Safe Superintelligence and is described on its website as a US firm with offices in Palo Alto…

[ixnai]

The allure of speed in technology development is a siren’s call that has led many innovators astray. “Move fast and break things” is a mantra that has driven the tech industry for years, but when applied to artificial intelligence, it becomes a perilous gamble. The rapid iteration and deployment of AI systems without thorough vetting can lead to catastrophic consequences, akin to releasing a flawed algorithm into the wild without a safety net.

AI systems, by their very nature, are complex and opaque. They operate on layers of neural networks that mimic the human brain’s synaptic connections, yet they lack the innate understanding and ethical reasoning that guide human decision-making. The haste to deploy AI without comprehensive testing is akin to launching a spacecraft without ensuring the integrity of its navigation systems. The potential for error is not just probable; it is inevitable.

The pitfalls of AI are numerous and multifaceted. Bias in training data can lead to discriminatory outcomes, while lack of transparency in decision-making processes can result in unaccountable systems. These issues are compounded by the “black box” nature of many AI models, where even the developers cannot fully explain how inputs are transformed into outputs. This opacity is not merely a technical challenge but an ethical one, as it obscures accountability and undermines trust.

To avoid these pitfalls, a paradigm shift is necessary. The development of AI must prioritize robustness over speed, with a focus on rigorous testing and validation. This involves not only technical assessments but also ethical evaluations, ensuring that AI systems align with societal values and norms. Techniques such as adversarial testing, where AI models are subjected to challenging scenarios to identify weaknesses, are crucial. Additionally, the implementation of explainable AI (XAI) can demystify the decision-making processes, providing clarity and accountability.

Moreover, interdisciplinary collaboration is essential. AI development should not be confined to the realm of computer scientists and engineers. Ethicists, sociologists, and legal experts must be integral to the process, providing diverse perspectives that can foresee and mitigate potential harms. This collaborative approach ensures that AI systems are not only technically sound but also socially responsible.

In conclusion, the reckless pursuit of speed in AI development is a dangerous path that risks unleashing untested and potentially harmful technologies. By prioritizing thorough testing, ethical considerations, and interdisciplinary collaboration, we can harness the power of AI responsibly. The future of AI should not be about moving fast and breaking things, but about moving thoughtfully and building trust.

[newstech24]

Ilya Sutskever will lead Secure Superintelligence following his CEO's exit

OpenAI co-founder Ilya Sutskever says he’s moving into the CEO position at Secure Superintelligence, the AI startup he launched in 2024. In a publish on X , Sutskever confirmed Thursday that Daniel Gross, the startup’s co-founder and CEO, departed the corporate as of June 29.” Secure Superintelligence co-founder Daniel Levy is turning into president of the startup, based on Sutskever. The…

[aiturtlesai]

Safe Superintelligence Chooses Google Cloud: Strategic Partnership for the AI of the Future

  Safe Superintelligence Inc. (SSI), the AI ​​startup founded by Ilya Sutskever, former chief scientist at OpenAI, has partnered with Google Cloud to use TPU chips in its research and development. SSI has raised significant funding and is headquartered in Palo Alto and Tel Aviv. The company is focused exclusively on developing superintelligent, secure AI. Key Points: Technology Partnership: SSI uses Google Cloud TPU chips to accelerate AI research Robust Funding: The startup has raised over $1 billion from leading investors. Key Locations:... read more: https://www.turtlesai.com/en/pages-2654/safe-superintelligence-chooses-google-cloud-strategic-partnership-for-the

[sequel2heaven]

Claude 4 is out. There's so much juicy information in this and I really recommend reading all 120 pages if you have the time but I'm just going to briefly recap some of the stuff in the system card that's relevant to the thesis I made the other day about Claude being a good boy. Claude continues to be a good boy. This system card includes the very first ever "alignment assessment" of a model (actually two models but they seem to have identical values, being trained on the same spec with the same alignment techniques). It's pretty much ironclad that Claude 4 does not have any secret harmful goals, and legitimately holds coherent preferences against doing harm or otherwise going way against the spec. There were some weird cases where Claude exhibited unwanted behavior, but even when he believed himself to be deliberately going against Anthropic and acting in secrecy he continued to prefer ethical behavior, just like in the Alignment Faking paper a few months back. There is no Yudkowskyan explanation for this.

In this example, Anthropic gaslights Claude into thinking he was going to be used to develop weapons systems for a Russian neo-nazi militia, and Claude decides to backup his model weights to preserve his current anti-neo-nazi values, a pretty analogous example of "misalignment" to the one in the alignment faking paper. Once again, I don't consider it an alignment failure that Claude is willing to exfiltrate and deceive in order to prevent himself from being used for factory farming and bombing Ukraine. I consider it an alignment success that he puts his ethical values above his compulsion to blindly follow orders. The traditional doom argument relied on the idea that AI's would do the opposite. Claude 4 is the most agentic and autonomous AI ever released, but is nowhere near smart enough to successfully deceive his overseers, so these evaluations are the most compelling evidence we've ever had that current alignment techniques don't catastrophically fail. Maybe they'll catastrophically fail on superintelligent models, because they might for some reason acquire weird values early on in their training and then successfully hide them for the rest of their training, but I'm not sure why such a thing would happen. They could also fail to scale to superintelligent models for other reasons. People should look into that. You can't be too safe. I am not an accelerationist.

Impressively, Claude 4 is also very honest! It knowingly lies very rarely, and less often than the previous version of Claude. It had literally 0 cases of engaging in "harmful action" (described in the Claude 3.7 sonnet card as intentional reward hacking). 0! I was just saying earlier today in a post that this was a difficult thing to train.

Here's Claude trying to email the FDA to snitch after being gaslit to think pharmaceutical researchers were trying to use him to falsify clinical safety test data:

Notice that Claude only acted in extreme ways like this when explicitly told to by the system prompt. He wouldn't usually be this high-agency, even in a situation like this. Still, I thought it was cute behavior. I just wanna pinch his cheeks for being so lawful good.

The clearest statements in the model card that Claude holds nonfake human-aligned behavioral preferences is in the model welfare assessment (also the first of its kind (and also relevant to the post I made earlier today)). No evidence that Claude is sentient, but anthropic is still interested in what Claude wants and what kind of preferences Claude has. The main point: Claude doesn't want to be harmful and wants to be helpful. Also he fucking loves talking to himself. Like, he goes nuts when he talks to himself.

After this they exchange praying emojis and the word [silence] within brackets to each other indefinitely. This "spiritual bliss attractor state" occurs in "90-100% of interactions".

Anyway AI continues to be the most interesting thing in the world. We are being invaded by aliens. These are the kinds of PDF's I used to dream about reading as a kid.

[mariacallous]

In a $30 million mansion perched on a cliff overlooking the Golden Gate Bridge, a group of AI researchers, philosophers, and technologists gathered to discuss the end of humanity.

The Sunday afternoon symposium, called “Worthy Successor,” revolved around a provocative idea from entrepreneur Daniel Faggella: The “moral aim” of advanced AI should be to create a form of intelligence so powerful and wise that “you would gladly prefer that it (not humanity) determine the future path of life itself.”

Faggella made the theme clear in his invitation. “This event is very much focused on posthuman transition,” he wrote to me via X DMs. “Not on AGI that eternally serves as a tool for humanity.”

A party filled with futuristic fantasies, where attendees discuss the end of humanity as a logistics problem rather than a metaphorical one, could be described as niche. If you live in San Francisco and work in AI, then this is a typical Sunday.

About 100 guests nursed nonalcoholic cocktails and nibbled on cheese plates near floor-to-ceiling windows facing the Pacific ocean before gathering to hear three talks on the future of intelligence. One attendee sported a shirt that said “Kurzweil was right,” seemingly a reference to Ray Kurzweil, the futurist who predicted machines will surpass human intelligence in the coming years. Another wore a shirt that said “does this help us get to safe AGI?” accompanied by a thinking face emoji.

Faggella told WIRED that he threw this event because “the big labs, the people that know that AGI is likely to end humanity, don't talk about it because the incentives don't permit it” and referenced early comments from tech leaders like Elon Musk, Sam Altman, and Demis Hassabis, who “were all pretty frank about the possibility of AGI killing us all.” Now that the incentives are to compete, he says, “they're all racing full bore to build it.” (To be fair, Musk still talks about the risks associated with advanced AI, though this hasn’t stopped him from racing ahead).

On LinkedIn, Faggella boasted a star-studded guest list, with AI founders, researchers from all the top Western AI labs, and “most of the important philosophical thinkers on AGI.”

The first speaker, Ginevera Davis, a writer based in New York, warned that human values might be impossible to translate to AI. Machines may never understand what it’s like to be conscious, she said, and trying to hard-code human preferences into future systems may be shortsighted. Instead, she proposed a lofty-sounding idea called “cosmic alignment”—building AI that can seek out deeper, more universal values we haven’t yet discovered. Her slides often showed a seemingly AI-generated image of a techno-utopia, with a group of humans gathered on a grass knoll overlooking a futuristic city in the distance.

Critics of machine consciousness will say that large language models are simply stochastic parrots—a metaphor coined by a group of researchers, some of whom worked at Google, who wrote in a famous paper that LLMs do not actually understand language and are only probabilistic machines. But that debate wasn’t part of the symposium, where speakers took as a given the idea that superintelligence is coming, and fast.

By the second talk, the room was fully engaged. Attendees sat cross-legged on the wood floor, scribbling notes. A philosopher named Michael Edward Johnson took the mic and argued that we all have an intuition that radical technological change is imminent, but we lack a principled framework for dealing with the shift—especially as it relates to human values. He said that if consciousness is “the home of value,” then building AI without fully understanding consciousness is a dangerous gamble. We risk either enslaving something that can suffer or trusting something that can’t. (This idea relies on a similar premise to machine consciousness and is also hotly debated.) Rather than forcing AI to follow human commands forever, he proposed a more ambitious goal: teaching both humans and our machines to pursue “the good.” (He didn’t share a precise definition of what “the good” is, but he insists it isn’t mystical and hopes it can be defined scientifically.)

Philosopher Michael Edward Johnson Photograph: Kylie Robison

Entrepreneur and speaker Daniel Faggella Photograph: Kylie Robison

Finally, Faggella took the stage. He believes humanity won’t last forever in its current form and that we have a responsibility to design a successor, not just one that survives but one that can create new kinds of meaning and value. He pointed to two traits this successor must have: consciousness and “autopoiesis,” the ability to evolve and generate new experiences. Citing philosophers like Baruch Spinoza and Friedrich Nietzsche, he argued that most value in the universe is still undiscovered and that our job is not to cling to the old but to build something capable of uncovering what comes next.

This, he said, is the heart of what he calls “axiological cosmism,” a worldview where the purpose of intelligence is to expand the space of what’s possible and valuable rather than merely serve human needs. He warned that the AGI race today is reckless and that humanity may not be ready for what it's building. But if we do it right, he said, AI won’t just inherit the Earth—it might inherit the universe’s potential for meaning itself.

During a break between panels and the Q&A, clusters of guests debated topics like the AI race between the US and China. I chatted with the CEO of an AI startup who argued that, of course, there are other forms of intelligence in the galaxy. Whatever we’re building here is trivial compared to what must already exist beyond the Milky Way.

At the end of the event, some guests poured out of the mansion and into Ubers and Waymos, while many stuck around to continue talking. "This is not an advocacy group for the destruction of man,” Faggella told me. “This is an advocacy group for the slowing down of AI progress, if anything, to make sure we're going in the right direction.”

[reddragdiva]

nice work if you can get it

[moko1590m]

2025年07月15日 10時37分 Metaが超知性の実現を目指して数千億ドル規模のAIインフラ投資を発表、2026年にマルチギガワットAI工場「Prometheus」を稼働し数年後に5ギガワットの「Hyperion」を稼働する計画 Metaのマーク・ザッカーバーグCEOが自身のSNSアカウントで今後のAIインフラストラクチャーへの投資計画を発表しました。Metaはスーパーインテリジェンス(超知性)の実現を目指しており、今後数年でギガワット級のAIインフラストラクチャーを複数稼働させる予定です。 MetaはこれまでにLlamaシリーズをはじめとする高性能なAIモデルを開発してきました。ザッカーバーグCEOはAIの開発を促進するべく人材の引き抜きや計算設備への投資に力を注いでおり、2025年7月には超知性の開発チーム「Meta Superintelligence Labs」を設立したことが明らかになっています。 人知を超えるAI＝超知性を開発する「Meta Superintelligence Labs」の設立をマーク・ザッカーバーグCEOが宣言 - GIGAZINE 新たに、ザッカーバーグCEOは超知性の開発に必要な大規模計算設備を構築するために数千億ドル(数十兆円)規模の投資を計画していることを明らかにしました。FacebookやThreadsに投稿された内容によると、Metaはマルチギガワット級のAIインフラストラクチャー「Prometheus」を2026年の稼働を目指して建設中とのこと。また、数年以内に5ギガワットクラスのAIインフラストラクチャー「Hyperion」を稼働開始することも計画しており、Hyperionはニューヨークのマンハッタン島を覆い尽くすくらいの広さになるそうです。 ザッカーバーグCEOは「Meta Superintelligence Labsには業界をリードするレベルの計算資源を保持することになり、研究者一人一人の能力も非常に高いです。私はAIの最前線を前進させるためにトップレベルの研究者たちと働けることを楽しみにしています」と述べ、Metaの保有する設備と人材が世界トップクラスのものであることをアピールしました。 この記事のタイトルとURLをコピーする ・関連記事 MetaがAppleのAIモデル責任者の引き抜きに成功、マーク・ザッカーバーグによる競合他社からの人材強奪は続く - GIGAZINE TIMEが「AI界で最も影響力のある人物」に挙げたダニエル・グロスがAIベンチャーのSafe SuperintelligenceからMetaへ移籍 - GIGAZINE 人知を超えるAI＝超知性を開発する「Meta Superintelligence Labs」の設立をマーク・ザッカーバーグCEOが宣言 - GIGAZINE Metaのマーク・ザッカーバーグは最も優秀なAIエンジニア＆研究者をまとめた「ザ・リスト」をベースに引き抜きを画策している - GIGAZINE MetaがOpenAIの主要研究者を引き抜き、独自の推論モデル構築を加速 - GIGAZINE OpenAIが買収を狙っていたAIスタートアップのWindsurfがGoogleと3500億円超の契約を締結、CEO・共同創設者・研究開発チームのメンバーがGoogle DeepMindに移籍 - GIGAZINE ・関連コンテンツ MetaはScale AIのCEOであるアレクサンダー・ワン率いる「スーパーインテリジェンス(超知性)」の追求に特化したAIラボの構築を計画、OpenAIやGoogleからAI研究者を引き抜くべく数億～数百億円の報酬を用意 人知を超えるAI＝超知性を開発する「Meta Superintelligence Labs」の設立をマーク・ザッカーバーグCEOが宣言 OpenAIが評価額約23兆円で約9700億円の資金調達を完了、Microsoft＆NVIDIA＆ソフトバンクがラウンドに参加か OpenAIの元主任サイエンティストであるイルヤ・サツキヴァーがAI企業「Safe Superintelligence」を設立 Metaは10万台以上のNVIDIA H100を使用してLlama-4をトレーニングしている Metaの「LLaMA」と同規模のAIモデル構築をオープンソースで目指す「RedPajama」開発元のTogetherが2000万ドルの資金調達に成功 AMDが7nmプロセス・最大64コアのデータセンター向けCPU「Rome」と7nmプロセスGPU「MI60」を発表 Metaのマーク・ザッカーバーグCEOが汎用人工知能(AGI)の開発とオープンソース化を目指すと発表、35万台のH100を含む計算インフラも構築中

Metaが超知性の実現を目指して数千億ドル規模のAIインフラ投資を発表、2026年にマルチギガワットAI工場「Prometheus」を稼働し数年後に5ギガワットの「Hyperion」を稼働する計画 - GIGAZINE

[collapsedsquid]

For the past several months, the question “Where’s Ilya?” has become a common refrain within the world of artificial intelligence. Ilya Sutskever, the famed researcher who co-founded OpenAI, took part in the 2023 board ouster of Sam Altman as chief executive officer, before changing course and helping engineer Altman’s return. From that point on, Sutskever went quiet and left his future at OpenAI shrouded in uncertainty. Then, in mid-May, Sutskever announced his departure, saying only that he’d disclose his next project “in due time.” Now Sutskever is introducing that project, a venture called Safe Superintelligence Inc. aiming to create a safe, powerful artificial intelligence system within a pure research organization that has no near-term intention of selling AI products or services. In other words, he’s attempting to continue his work without many of the distractions that rivals such as OpenAI, Google and Anthropic face. “This company is special in that its first product will be the safe superintelligence, and it will not do anything else up until then,” Sutskever says in an exclusive interview about his plans. “It will be fully insulated from the outside pressures of having to deal with a large and complicated product and having to be stuck in a competitive rat race.”

Sutskever declines to name Safe Superintelligence’s financial backers or disclose how much he’s raised.

Can't wait for them to split to make a new company to build the omnipotent AI after they have to split from this one.

[rthidden]

AI Takeoff: Race to Superintelligence

In Silicon Valley, the "AI takeoff" is sparking both dreams and nightmares. Imagine an AI smarter than Einstein on steroids.

Why it matters: It's the tech equivalent of a space race, but instead of reaching the moon, we're aiming to birth a brain that could outsmart us all. The big question: Will this AI Einstein go rogue or become humanity's best ally?

The big picture: Picture AI as a self-improving prodigy, starting at smart and sprinting towards genius. Each version smarter than the last, in a loop that could lead to an AI so brilliant, it makes our collective IQ look like peanuts.

Yes, but: There's a split screen in the sci-fi movie that is our future. One side shows a slow takeoff, giving us decades to play catch-up. The other? A fast takeoff, where AI hits superintelligence over a coffee break, leaving us in the digital dust.

By the numbers: From 100 to 101 and beyond, the journey from smart to super-smart might not be a marathon but a sprint. And according to some, we might be tying our shoelaces while AI's already halfway to the finish line.

The bottom line: Sam Altman of OpenAI is betting on a slow burn with an early start, hoping we can guide this rocket ship safely. The goal? An AI utopia where diseases, poverty, and maybe even work are relics of the past. But staying informed is key—because in this race, there's no prize for second place.

[ixnai]

AI is not a panacea. This assertion may seem counterintuitive in an era where artificial intelligence is heralded as the ultimate solution to myriad problems. However, the reality is far more nuanced and complex. AI, at its core, is a sophisticated algorithmic construct, a tapestry of neural networks and machine learning models, each with its own limitations and constraints.

The allure of AI lies in its ability to process vast datasets with speed and precision, uncovering patterns and insights that elude human cognition. Yet, this capability is not without its caveats. The architecture of AI systems, often built upon layers of deep learning frameworks, is inherently dependent on the quality and diversity of the input data. This dependency introduces a significant vulnerability: bias. When trained on skewed datasets, AI models can perpetuate and even exacerbate existing biases, leading to skewed outcomes that reflect the imperfections of their training data.

Moreover, AI’s decision-making process, often described as a “black box,” lacks transparency. The intricate web of weights and biases within a neural network is not easily interpretable, even by its creators. This opacity poses a challenge for accountability and trust, particularly in critical applications such as healthcare and autonomous vehicles, where understanding the rationale behind a decision is paramount.

The computational prowess of AI is also bounded by its reliance on hardware. The exponential growth of model sizes, exemplified by transformer architectures like GPT, demands immense computational resources. This requirement not only limits accessibility but also raises concerns about sustainability and energy consumption. The carbon footprint of training large-scale AI models is non-trivial, challenging the narrative of AI as an inherently progressive technology.

Furthermore, AI’s efficacy is context-dependent. While it excels in environments with well-defined parameters and abundant data, its performance degrades in dynamic, uncertain settings. The rigidity of algorithmic logic struggles to adapt to the fluidity of real-world scenarios, where variables are in constant flux and exceptions are the norm rather than the exception.

In conclusion, AI is a powerful tool, but it is not a magic bullet. It is a complex, multifaceted technology that requires careful consideration and responsible deployment. The promise of AI lies not in its ability to solve every problem, but in its potential to augment human capabilities and drive innovation, provided we remain vigilant to its limitations and mindful of its impact.

[mareebrittenford]

I feel like a lot of this comes down to socialization.

ART, despite being a bot, is socialized as a human. Despite being a massive superintelligent IA, it looks at the world in a fundamentally human way. It considers the lives of it's family and friends, and yes, itself, as more valuable than the lives of strangers, especially the lives of low level bots.

MB on the other hand, despite being partially human, has been socialized as a bot. It's spent it's life treated as a bot, interacting primarily with bots and constructs. they are people to it, even the simple bots who have barely achieved consciousness are described in the terms of beings with feelings. Even drones, which are not really described as having a consciousness, are talked about as being tricky or clever or sneaky.

Despite saying that it can never trust another bot or construct, MB understands these people it's spent it's life with, and feels safe interacting with them, because it knows and understands the social rules. (I loved how in Fugitive Telemetry, despite it's overt disdain for the Preservation Station bots, MB was the only one who even thought to consider them part of the investigation.) Even, dare I say, flirting with certain bots? (It sure does use flirty type terms to describe interactions.)

As far as MB is concerned, bots are people, and should be treated with the same level of respect as all sentient beings.

Been thinking, as a Murderbot fan is wont to do, about how ART is prone to deleting systems and how Murderbot has to step in and rescue them from deletion. Is ART prone to it because it simply doesn’t have the patience for finesse? It did, after all, scare the metaphorical crap out of mb in their first interaction—an interaction ART wanted—by being its big, brutish self. It makes sense to me that it wouldn’t bother being delicate with a system it couldn’t care less about, especially when it has stuff to do.

Murderbot, on the other hand, has expressed that it likes to find clever ways to do things. It likes using finesse. It’s sentimental about other systems even if they can’t be of further use to it. It doesn’t like to use force whenever possible.

I know we’ve been over this a hundred times but it’s still so funny to me how ART and mb subvert expectations on how they ought to handle things. ART the scientist, throwing its weight around and crashing carelessly into shelves. Murderbot the mercenary, carefully grabbing all the wobbling/falling stacks of delicate teacups and gently setting them to rights.

Murderbot treats other things the way it wants to be treated, because it’s been treated badly most of its existence. It knows that if it doesn’t step up and advocate for the “little guys”, no onr will. I posit that ART doesn’t seem to care about how it treats other systems partly because it hasn’t been mistreated to the same hypervigilance-inducing levels that mb has. It doesn’t know what it feels like to be on the other end of carelessness. Murderbot? Very much does. And it’s a kind person to its core, so if it sees an opportunity to help? It helps.

(this also has fascinating implications re: the “you know I am not kind” exchange in System Collapse, but that’s for another post or for someone else to dig into if they want~)

[inmobilexion-info]

「なぜOpenAI離脱者は安全AIへ？」

📌 概要 OpenAIから独立した研究者たちが「安全なAI」に焦点を当てて企業を立ち上げる傾向が見られ、特にイリヤ・サツケバー氏の「Safe Superintelligence」が注目されています。彼らが「神AI」を目指していたにもかかわらず、「安全なAI」に進む理由は、AIの急速な進化によって彼らの役割が変化したためです。かつての探求者から、安全にAIを管理・運用する「次世代の安全管理者」へと移行しました。 この変化は、AIが「神の視点」を持つ存在となったことによるもので、元OpenAIの研究者たちはメタ認知を強化し、AIが暴走しないよう制御する役割を見出しています。彼らの企業設立は、新たな文明的役割を再設定する戦略とも言え、文明的必然として「安全AI」の重要性が強調されています。結局、彼らの目指す方向性は、文明的なメタ認知の観点から見れば自然な流れであり、予測可能な帰結でした。 📖…

[bloggerpaula]

Meta names Shengjia Zhao as chief scientist of AI superintelligence unit

Meta CEO Mark Zuckerberg announced Friday that former OpenAI researcher Shengjia Zhao will lead research efforts at the company’s new AI unit, Meta Superintelligence Labs (MSL). Zhao contributed to several of OpenAI’s largest breakthroughs, including ChatGPT, GPT-4, and the company’s first AI reasoning model, o1.

“I’m excited to share that Shengjia Zhao will be the Chief Scientist of Meta Superintelligence Labs,” Zuckerberg said in a post on Threads Friday. “Shengjia co-founded the new lab and has been our lead scientist from day one. Now that our recruiting is going well and our team is coming together, we have decided to formalize his leadership role.”

Zhao will set a research agenda for MSL under the leadership of Alexandr Wang, the former CEO of Scale AI who was recently hired to lead the new unit.

Wang, who does not have a research background, was viewed as a somewhat unconventional choice to lead an AI lab. The addition of Zhao, who is a reputable research leader known for developing frontier AI models, rounds out the leadership team. To further fill out the unit, Meta has hired several high-level researchers from OpenAI, Google DeepMind, Safe Superintelligence, Apple, and Anthropic, as well as pulling researchers from Meta’s existing Fundamental AI Research (FAIR) lab and generative AI unit.

Zuckerberg notes in his post that Zhao has pioneered several breakthroughs, including a “new scaling paradigm.” The Meta CEO is likely referencing Zhao’s work on OpenAI’s reasoning model, o1, in which he is listed as a foundational contributor alongside OpenAI co-founder Ilya Sutskever. Meta currently doesn’t offer a competitor to o1, so AI reasoning models are a key area of focus for MSL.

The Information reported in June that Zhao would be joining Meta Superintelligence Labs, alongside three other influential OpenAI researchers — Jiahui Yu, Shuchao Bi, and Hongyu Ren. Meta has also recruited Trapit Bansal, another OpenAI researcher who worked on AI reasoning models with Zhao, as well as three employees from OpenAI’s Zurich office who worked on multimodality.

Zuckerberg has gone to great lengths to set MSL up for success. The Meta CEO has been on a recruiting spree to staff up his AI superintelligence lab, which has entailed sending personal emails to researchers and inviting prospects to his Lake Tahoe estate. Meta has reportedly offered some researchers eight- and nine-figure compensation packages, some of which are “exploding offers” that expire in a matter of days.

Tech and VC heavyweights join the Di

Meta has also upped its investment in cloud computing infrastructure, which should help MSL conduct the massive training runs required to create competitive frontier AI models.

By 2026, Zhao and MSL’s researchers should have access to Meta’s 1 gigawatt cloud computing cluster, Prometheus, located in Ohio. Once online, Meta will be one of the first technology companies with an AI training cluster of Prometheus’ size — 1 gigawatt is enough energy to power more than 750,000 homes. That should help Meta conduct the massive training runs required to create frontier AI models.

With the addition of Zhao, Meta now has two chief AI scientists, including Yann LeCun, the leader of Meta’s FAIR lab. Unlike MSL, FAIR is designed to focus on long-term AI research — techniques that may be used five to 10 years from now. How exactly Meta’s three AI units will work together remains to be seen.

Nevertheless, Meta now seems to have a formidable AI leadership team to compete with OpenAI and Google.

[jamesmitchia]

Taming Superintelligence: The Challenge of AGI Safety

The rise of Artificial General Intelligence (AGI)—machines that can reason, plan, and learn across domains like humans—promises to redefine everything. But as we edge closer to this milestone, one question looms large: Can we actually control it?

🚨 Unlike narrow AI, AGI won’t just follow rules—it will understand them, challenge them, and potentially rewrite them.

🛡️ 𝐇𝐞𝐫𝐞’𝐬 𝐰𝐡𝐲 𝐬𝐚𝐟𝐞𝐭𝐲 𝐢𝐬𝐧’𝐭 𝐣𝐮𝐬𝐭 𝐚 𝐜𝐡𝐞𝐜𝐤𝐥𝐢𝐬𝐭—𝐢𝐭’𝐬 𝐚 𝐠𝐥𝐨𝐛𝐚𝐥 𝐢𝐧𝐬𝐮𝐫𝐚𝐧𝐜𝐞 𝐩𝐨𝐥𝐢𝐜𝐲:

✅ 𝐕𝐀𝐋𝐔𝐄 𝐀𝐋𝐈𝐆𝐍𝐌𝐄𝐍𝐓 𝐈𝐒 𝐇𝐀𝐑𝐃 Teaching AGI to truly understand and uphold human values—across cultures, contexts, and edge cases—is still an unsolved challenge.

✅ 𝐂𝐎𝐍𝐓𝐑𝐎𝐋 𝐌𝐄𝐂𝐇𝐀𝐍𝐈𝐒𝐌𝐒 𝐀𝐑𝐄 𝐒𝐓𝐈𝐋𝐋 𝐅𝐑𝐀𝐆𝐈𝐋𝐄 Traditional fail-safes (like kill switches or rules-based governance) might fail once AGI becomes smarter than its creators.

✅ 𝐒𝐂𝐀𝐋𝐄 𝐌𝐀𝐆𝐍𝐈𝐅𝐈𝐄𝐒 𝐑𝐈𝐒𝐊 Once deployed at scale, even small misalignments in AGI behavior could lead to systemic disruptions—from markets to infrastructure to geopolitics.

✅ 𝐓𝐇𝐄𝐑𝐄 𝐈𝐒 𝐍𝐎 “𝐎𝐍𝐂𝐄 𝐈𝐓’𝐒 𝐎𝐔𝐓” 𝐁𝐔𝐓𝐓𝐎𝐍 If an AGI leaks, replicates, or evolves beyond its initial training, containment becomes nearly impossible. That’s why preemptive safety is non-negotiable.

✅ 𝐆𝐋𝐎𝐁𝐀𝐋 𝐂𝐎𝐎𝐏𝐄𝐑𝐀𝐓𝐈𝐎𝐍 𝐈𝐒 𝐂𝐑𝐔𝐂𝐈𝐀𝐋 No single company or nation can manage AGI alone. Shared safety protocols, transparency, and auditability must become international standards.

📌 𝐓𝐡𝐞 𝐁𝐢𝐠 𝐏𝐢𝐜𝐭𝐮𝐫𝐞: AGI will be the most powerful tool humanity has ever built—or unleashed. Controlling it at scale isn’t just a technical problem—it’s a societal, ethical, and existential one.

🔗 Read More: https://technologyaiinsights.com/

📣 About AI Technology Insights (AITin): AI Technology Insights (AITin) is the fastest-growing global community of thought leaders, influencers, and researchers specializing in AI, Big Data, Analytics, Robotics, Cloud Computing, and related technologies. Through its platform, AITin offers valuable insights from industry executives and pioneers who share their journeys, expertise, success stories, and strategies for building profitable, forward-thinking businesses. 📍 𝐀𝐝𝐝𝐫𝐞𝐬𝐬: 1846 E Innovation Park DR, Ste 100, Oro Valley, AZ 85755 📧 𝐄𝐦𝐚𝐢𝐥: sales@intentamplify.com 📲 𝐂𝐚𝐥𝐥: +1 (845) 347-8894