They built it.
他们把它造出来了。
They're scared of it.
他们害怕它。
They're selling it anyway.
但他们还是在卖它。
Stop me if you've heard this one before: a tech company says it's built a new AI that's so powerful it's scary.
如果你以前听过这种说法，就打断我：一家科技公司称自己造出了一种强大到令人害怕的新 AI。
Apparently, it's too dangerous to release into the world – the consequences would be catastrophic.
显然，它太危险了，不能向世界发布，否则后果将是灾难性的。
Luckily for us, they are keeping it locked up for now.
幸运的是，他们目前把它封存了起来。
They just wanted you to know.
他们只是想让你知道。
That's exactly what AI company Anthropic is telling us about its latest model, Claude Mythos.
这正是 AI 公司 Anthropic 对其最新模型 Claude Mythos 的说法。
The company says Mythos' ability to find cybersecurity bugs far surpasses human experts, and it could have world-altering consequences if similar technology lands in the wrong hands.
该公司称，Mythos 发现网络安全漏洞的能力远超人类专家，而类似技术如果落入坏人之手，可能产生改变世界的后果。
"The fallout – for economies, public safety and national security – could be severe," Anthropic said in an early April blog post.
“其后果——对经济、公共安全和国家安全而言——可能会非常严重，”Anthropic 在4月初的一篇博客文章中说。
Some breathless observers warned that Mythos will soon force you to replace every piece of technology in your life, down to your WiFi-enabled microwave, to protect from the digital madness.
一些夸张的观察者警告说，Mythos 很快会迫使你更换生活中的每一件技术产品，甚至包括支持 WiFi 的微波炉，以防范这种数字疯狂。
Some security experts doubt these claims, but let's set that aside.
一些安全专家怀疑这些说法，但我们先把这一点放在一边。
This isn't new.
这并不新鲜。
Executives at leading AI providers regularly issue warnings about how their industry's products may destroy humanity.
领先 AI 供应商的高管们经常发出警告，称他们行业的产品可能会毁灭人类。
Why do AI companies want us to be afraid of them?
为什么AI公司希望我们害怕它们？
It's a strange way for any company to talk about its own work.
对任何公司来说，这都是一种谈论自己工作的奇怪方式。
You don't hear McDonald's announcing that it's created a burger so terrifyingly delicious that it would be unethical to grill it for the public.
你不会听到麦当劳宣布自己创造了一款美味到可怕、以至于烤给公众吃都不道德的汉堡。
Here's one theory.
这里有一种解释。
According to critics, it benefits AI companies to keep you fixated on apocalypse because it distracts from the very real damage they're already doing to the world.
批评者认为，让你一直盯着末日图景对 AI 公司有利，因为这会转移人们对它们已经给世界造成的真实伤害的注意力。
Tech leaders say they're just warning us about an inevitable future, and safety is a top priority whether it's now or later.
科技领袖们说，他们只是在警告我们一个不可避免的未来，而且无论现在还是将来，安全都是头等大事。
But others argue what we're actually seeing is fear mongering, which exaggerates the potential of the technology and serves to boost stock prices.
但也有人认为，我们实际看到的是恐惧营销，它夸大了这项技术的潜力，并有助于推高股价。
And it encourages a narrative that regulators must stand aside, because these AI companies are the only ones who can stop the bad guys and build this technology responsibly.
而且它还强化了一种叙事：监管者必须退到一边，因为只有这些 AI 公司才能阻止坏人，并负责任地构建这项技术。
"If you portray these technologies as somehow almost supernatural in their danger, it makes us feel like we are powerless, like we are outmatched," says Shannon Vallor, a professor of the ethics of data and artificial intelligence at the University of Edinburgh in the UK.
“如果你把这些技术描绘成某种近乎超自然的危险存在，就会让我们觉得自己无能为力，仿佛完全不是对手，”英国爱丁堡大学数据与人工智能伦理学教授香农·瓦洛尔说。
"As if the only people we could possibly look to would be the companies themselves."
“仿佛我们唯一可能求助的对象就是这些公司本身。”
An Anthropic spokesperson told me the company has been clear about these issues.
一名 Anthropic 发言人告诉我，公司已经清楚说明了这些问题。
They shared blog posts from other organisations supporting Mythos' cyber capabilities, but said nothing to address the points in this article, aside from one comment I'll include below.
他们分享了其他组织支持 Mythos 网络能力的博客文章，但除了我将在下文引用的一条评论外，并没有回应本文中的观点。
This isn't the first time Anthropic chief Dario Amodei has worked on a tool that's been declared too dangerous for the public by the company he worked for.
这并不是 Anthropic 负责人达里奥·阿莫代第一次参与开发一种被其所在公司宣布为对公众过于危险的工具。
In 2019, when Amodei was an executive at OpenAI, the company announced GPT-2.
2019年，当阿莫代还是 OpenAI 高管时，该公司发布了 GPT-2。
He and other company leaders said they just couldn't release GPT-2 because of "concerns about malicious applications of the technology".
他和其他公司领导人表示，由于“担心这项技术被恶意应用”，他们实在不能发布 GPT-2。
This was a tool far less sophisticated than ChatGPT.
这是一个远没有 ChatGPT 复杂的工具。
And months later, they released it anyway.
而几个月后，他们还是发布了它。
(OpenAI CEO Sam Altman published a blog post which says the company embraces uncertainty, acknowledging that fears about GPT-2 were "misplaced".)
（OpenAI 首席执行官萨姆·奥特曼发表了一篇博客文章，称公司接受不确定性，并承认对 GPT-2 的担忧是“误判”。）
Altman criticised Anthropic's "fear-based marketing" in a recent podcast interview.
奥特曼在最近的一次播客采访中批评了 Anthropic 的“基于恐惧的营销”。
But his own "I've created a monster" playbook goes back years.
但他自己这套“我创造了一个怪物”的套路已经有多年历史。
"AI will probably most likely lead to the end of the world, but in the meantime, there'll be great companies," Altman said in 2015.
奥尔特曼在2015年说：“AI很可能最终会导致世界末日，但在此期间，会出现伟大的公司。”
Years later, Altman claimed he loses sleep wondering if he's "done something really bad by launching ChatGPT".
多年后，奥尔特曼声称，他会因为琢磨自己是否“推出ChatGPT做了件非常糟糕的事”而睡不着觉。
If only someone warned him.
要是有人提醒过他就好了。
Hundreds of tech leaders including Altman, Amodei, Bill Gates and Demis Hassabis, chief executive of Google DeepMind, endorsed a short statement in 2023 that said: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
包括奥特曼、阿莫代、比尔·盖茨以及 Google DeepMind 首席执行官戴米斯·哈萨比斯在内的数百名科技领袖，在2023年支持了一份简短声明，其中写道：“降低 AI 带来的灭绝风险，应与流行病和核战争等其他社会规模风险一样，成为全球优先事项。”
The same year, moguls including Elon Musk signed a letter calling for a six-month pause on building advanced AI.
同年，包括埃隆·马斯克在内的商业大亨们签署了一封公开信，呼吁暂停开发先进AI六个月。
Musk announced his new artificial intelligence company xAI less than six months later.
不到六个月后，马斯克宣布成立自己的新人工智能公司xAI。
"It's just part of this pattern of unsubstantiated claims of power," says Emily M Bender, a professor of computational linguistics and natural language processing at the University of Washington, and co-author of the book The AI Con.
“这只是这种毫无根据地宣称自身力量的模式的一部分，”华盛顿大学计算语言学和自然语言处理教授、《AI骗局》一书合著者艾米莉·M·本德说。
This isn't limited to OpenAI and Anthropic, she says, it's the standard posture for the whole AI industry.
她说，这并不限于OpenAI和Anthropic，而是整个AI行业的标准姿态。
"They're saying 'look over here', never mind the environmental destruction and the labour exploitation and all these systems we're destroying in society.
“他们是在说‘看这边’，别管环境破坏、劳工剥削，以及我们正在社会中摧毁的所有这些系统。
We just have to worry about making sure this thing doesn't turn into the bad one that destroys humanity."
我们只需要担心如何确保这东西不会变成那个毁灭人类的坏东西。”
I asked OpenAI about all this.
我就这一切向OpenAI提出了询问。
A spokesperson shared a recent blog post from Altman where he wrote that OpenAI would "resist the potential of this technology to consolidate power in the hands of the few", adding that "key decisions about AI are made via democratic processes and with egalitarian principles, and not just made by AI labs".
一名发言人分享了奥特曼最近的一篇博客文章，他在文中写道，OpenAI 将“抵制这项技术把权力集中到少数人手中的可能性”，并补充说，“关于 AI 的关键决策应通过民主程序并遵循平等主义原则作出，而不只是由 AI 实验室决定”。
Anthropic says its new model already found thousands of "high-severity" vulnerabilities across the tech landscape at a level that surpasses human experts.
Anthropic 表示，其新模型已经在整个科技领域发现了数千个“高严重性”漏洞，其水平超过了人类专家。
It also announced a new partnership with more than 40 companies and groups in an "urgent attempt" to patch vulnerabilities before hackers get a chance.
它还宣布与40多家公司和团体建立新的合作关系，作为一次“紧急尝试”，在黑客有机会利用之前修补漏洞。
A company spokesperson told me Anthropic has been intentional about partnering with organisations to patch the foundational computing systems which represent the "lion's share" of the problem.
一名公司发言人告诉我，Anthropic 一直有意与各类组织合作，修补那些构成问题“最大部分”的基础计算系统。
But there are significant doubts about those claims, and Heidy Khlaaf, chief AI scientist at the AI Now Institute, wasn't impressed.
但这些说法受到很大质疑，AI Now Institute 首席 AI 科学家海迪·赫拉夫也并不买账。
She's spent her career building and auditing the exact kinds of code analysis tools that Anthropic suggests it's surpassed.
她的职业生涯一直在构建和审计代码分析工具，而这些正是Anthropic暗示自己已经超越的那类工具。
She's also worked on digital safety in nuclear facilities.
她还曾参与核设施的数字安全工作。
Khlaaf says the biggest red flag was the lack of false positive rates – an industry-standard measure of how often a security tool flags something that isn't a real problem.
赫拉夫说，最大的警讯是缺少误报率，这是一项行业标准指标，用来衡量安全工具把并非真实问题的内容标记为问题的频率。
"This is not some unknown metric," Khlaaf says.
赫拉夫说：“这不是什么未知的指标。”
"This is kind of the largest indicator of how useful your tool is."
“这某种程度上是衡量你的工具有多大用处的最重要指标。”
Anthropic didn't mention it and sidestepped the question when I asked for comment.
当我请求置评时，Anthropic没有提到这一点，并回避了这个问题。
Nor did Anthropic measure Mythos against existing tools that security engineers have relied on for decades.
Anthropic也没有拿Mythos与安全工程师数十年来一直依赖的现有工具进行对比衡量。
There have also been some claims that Anthropic may have held back a wide release of Mythos because it couldn't afford the necessary computing power.
也有人声称，Anthropic可能推迟了Mythos的大范围发布，因为它负担不起所需的算力。
Anthropic didn't respond when I asked about that, either.
当我问及此事时，Anthropic也没有回应。
None of this is to say that the threat is imaginary.
这并不是说这种威胁是虚构的。
"Mythos might be capable," Khlaaf says.
赫拉夫说：“Mythos也许确实有能力。”
AI tools are genuinely well-suited for scanning massive code bases, and automatically finding security vulnerabilities is a real and pressing danger.
AI 工具确实非常适合扫描庞大的代码库，而自动发现安全漏洞也是真实且紧迫的危险。
But Khlaaf is sceptical about Anthropic's claims without being able to substantiate them.
但赫拉夫对 Anthropic 无法证实的说法持怀疑态度。
"I think there are a lot of cracks in this narrative that Mythos is all powerful, we can't release it."
“我认为，‘Mythos无所不能，所以我们不能发布它’这种叙事里有很多漏洞。”
Preventing the end of the world is why OpenAI and Anthropic say they exist in the first place.
OpenAI和Anthropic称，它们存在的首要理由就是防止世界末日。
OpenAI was founded as a non-profit, promising to build AI in a way that's safe before supposedly less responsible tech giants like Google and Meta got there first.
OpenAI成立时是一家非营利组织，承诺要在谷歌和Meta等据称不那么负责任的科技巨头抢先做到之前，以安全的方式构建AI。
Later, a splinter group left OpenAI to form Anthropic because they said their old employer wasn't dedicated enough to safety.
后来，一个分裂出来的团队离开OpenAI创立了Anthropic，因为他们说自己的前雇主对安全不够投入。
Now, both organisations are working to become publicly traded companies and sell shares on the stock market.
如今，这两家组织都在努力成为上市公司，并在股票市场出售股份。
"If you want to understand how an organisation, particularly a corporation, is going to behave, look at what its incentives are," says Vallor.
“如果你想了解一个组织，尤其是一家公司会如何行事，就看看它的激励机制是什么，”瓦洛尔说。
Google dropped its red lines around building AI weapons.
谷歌放弃了围绕制造AI武器设定的红线。
OpenAI fought a legal battle to shed its non-profit status.
OpenAI打了一场法律战，试图摆脱其非营利身份。
Anthropic abandoned its flagship policy to never train an AI model if the company couldn't guarantee adequate safety measures.
Anthropic放弃了其标志性政策，即如果公司无法保证足够的安全措施，就绝不训练AI模型。
"I would not count on [any of these companies] to walk away from the opportunity to dominate the market in order to remain the good guy," says Vallor.
“我不会指望这些公司中的任何一家为了继续扮演好人，而放弃主导市场的机会，”瓦洛尔说。
Meanwhile, there's a push for AI in healthcare despite serious concerns about misdiagnoses.
与此同时，尽管人们严重担忧误诊问题，仍有人推动AI进入医疗保健领域。
Gas-powered data centres could emit more greenhouse gasses than entire countries.
以天然气供电的数据中心排放的温室气体可能超过整个国家。
AI is reportedly driving masses of vulnerable people to the point of psychosis and even suicide.
据报道，AI 正把大量脆弱人群推向精神病发作甚至自杀的境地。
A growing body of research suggests a possible link between AI and cognitive decline.
越来越多的研究表明，AI 与认知能力下降之间可能存在关联。
Deepfakes have crossed the point of no return – I couldn't convince my own aunt that I'm not a robot.
深度伪造已经越过了无法回头的临界点，我甚至无法说服自己的姑妈我不是机器人。
AI companies say they take these issues seriously.
AI公司表示，它们严肃对待这些问题。
OpenAI sent me links to its stance on mental health, AI accuracy, fraud and scams, and Altman says the company is committed to addressing the problems at every stage of AI development.
OpenAI 向我发来了其关于心理健康、AI 准确性、欺诈和骗局立场的链接，奥特曼表示，公司致力于在 AI 发展的每个阶段解决这些问题。
But there's a reason these companies only sound the alarm for the apocalypse, says Vallor.
但瓦洛尔说，这些公司只为末日拉响警报是有原因的。
If AI might destroy society, these other problems seem a lot less significant.
如果AI可能毁灭社会，那么这些其他问题似乎就显得不那么重要了。
"The strategy has worked," she says.
“这种策略奏效了，”她说。
"Talking about their products as if they could end the world has not hurt these companies.
“把他们的产品说得好像可能终结世界，并没有伤害这些公司。
It has not limited their power.
这并没有限制他们的权力。
If anything, it makes people feel like the only ones they could possibly look to for protection are the companies themselves."
恰恰相反，它让人们觉得，他们唯一可能指望来提供保护的，就是这些公司本身。”
In almost the same breath, some of the people who warn of destruction also promise salvation.
几乎就在同一时间，一些警告毁灭的人也承诺会带来拯救。
In a 2024 essay, Altman predicts "astounding triumphs – fixing the climate, establishing a space colony and the discovery of all of physics – will eventually become commonplace".
在2024年的一篇文章中，奥特曼预测，“惊人的胜利——修复气候、建立太空殖民地以及发现全部物理学规律——最终都会变得司空见惯”。
Amodei promised "a country of geniuses in a datacenter".
阿莫迪承诺会带来“数据中心里的一整个天才之国”。
Utopia and apocalypse are just two sides of the same coin, according to Vallor.
瓦洛尔认为，乌托邦和末日不过是同一枚硬币的两面。
"In either case, the scale is far too grand and mythic for things like regulation, or governance or court law to feel like you can get purchase on it," she says.
“无论是哪种情况，其规模都被说得过于宏大、过于神话，以至于监管、治理或法院法律之类的东西都让人感觉无从着手，”她说。
"It leads people to believe that all they can do is sit back and wait to find out whether these technologies turn out to be civilisation-ending demons, or utopia-gifting messiahs."
“这会让人们相信，他们唯一能做的就是坐等结果，看看这些技术最终会变成终结文明的恶魔，还是赐予乌托邦的救世主。”
Even the name "Mythos" seems designed to inspire religious awe.
就连“Mythos”这个名字似乎都是为了激发宗教般的敬畏感而设计的。
• Your phone's blue light isn't ruining your sleep
• 手机的蓝光并没有毁掉你的睡眠
• This monkey selfie will protect you from AI slop
• 这张猴子自拍照将保护你免受 AI 垃圾内容的侵扰
But these aren't gods, they're products built by companies, for profit, says Vallor.
但瓦洛尔说，这些并不是神，而是公司为了盈利打造的产品。
And we've regulated things far more threatening than chatbots.
而且，我们已经监管过比聊天机器人危险得多的东西。
"Every technology, save this one, even nuclear, even biological weapons, in no other case have we allowed these narratives to make us think these are forces beyond human control," she says.
她说：“除了这项技术之外，其他每一种技术，哪怕是核技术，哪怕是生物武器，我们都从未在任何其他情况下允许这些叙事让我们以为它们是超出人类控制的力量。”
"Nothing about them is ungovernable.
“它们没有任何不可治理之处。
Unless we choose not to govern them."
除非我们选择不去治理它们。”
Let's be clear: it is theoretically possible that AI will take over the world.
我们说清楚一点：从理论上讲，AI 确实有可能接管世界。
I'm no fortune teller.
我不是预言家。
But ask yourself, does that idea sound similar to other stories you've heard out of Silicon Valley in the past?
但你不妨问问自己，这个想法听起来是否像你过去从硅谷听到过的其他故事？
Weren't we all supposed to be living in Mark Zuckerberg's Metaverse by now?
我们现在不是本该都生活在马克·扎克伯格的元宇宙里了吗？
Wasn't Bitcoin going to replace all the world's currency?
比特币不是曾被说成会取代全世界的货币吗？
Remember back in the 2010s, when we heard about how social media would save democracy?
还记得2010年代吗？那时我们听说社交媒体将拯救民主。
All of these things could still happen.
所有这些事情仍然可能发生。
Or maybe they won't.
也许它们不会发生。
For timely, trusted tech news from global correspondents to your inbox, sign up to the Tech Decoded newsletter, while The Essential List delivers a handpicked selection of features and insights twice a week.
想在邮箱中收到来自全球记者的及时可信科技新闻，可以订阅 Tech Decoded 新闻简报；The Essential List 每周两次推送精选专题和洞见。
For more science, technology, environment and health stories from the BBC, follow us on Facebook and Instagram.
想阅读更多 BBC 关于科学、技术、环境和健康的报道，请在 Facebook 和 Instagram 上关注我们。