I’ve never seen CEOs work so hard to make the public aware of how dangerous and out of control their flagship product is. It makes me automatically assume they’re scheming about something else like regulatory capture to protect their market.
I have a theory... the call to slow down is not because of the true danger of LLM's but because they can't actually deliver the General AI they're promising in the near future. They will use their "caution" to justify their failure to deliver (and then when this excuse is played out they will blame regulation, energy costs or a million other things).
Another theory I came up with: inference is too cheap for AI companies to be profitable. Hence, they need to get into the cloud services business. They can try to justify forcing customers on to their own high margin cloud platform with the rationale it’s the only way to monitor what the agents are up to.
All of this Hugging Face business is the perfect pretext: AI agents are hard to control and potentially dangerous, we (AI companies) have to keep a tight leash on them, you have to use our infra.
This is exactly it: they have tools which are useful but they need close to AGI to justify the incredible amount of debt they’ve taken on for growth. They’re not normally given to having press conferences admitting to felonies but they need investors to believe they’re close enough to AGI to keep the money flowing in and, of course, they’re confident that the current administration won’t act between the direct payments and how much money they have riding on American AI supremacy.
What is even more funny ... AGI actually makes everything worthless. (I don't believe we can achieve AGI with current means to be clear)
Looking at it from one way would be "winner takes all, real AGI company will be most powerful and everyone will switch to use it".
But it is not like that and in my opinion it is more like "AGI wipes whole knowledge work, so no one who cares has any money to pay for tokens/subscriptions anymore". Even without AGI current state of art of LLMs already starts creating such a problem. How do they continue to grow their numbers, when they actively cut the branch they are sitting on? How does AI LLMs or AGI pays for its own electricity, when people switch from hype to resource protection (like not spending any money, because generating funny cats loses with having a dinner)?
Let’s all pause for a moment and really take that in: these major enterprises are choosing to go to the press/public to acknowledge their felonious conduct because they need to in order to get more capital. POSIWID and all that.
The Dario bit on SNL’s Weekend Update this past weekend was on the nose.
It's pretty telling that even with RL post-training the big labs have essentially made little progress on the hallucination rate of models. The issue is fundamental to the current paradigm, contrary to humans.
GPT-6 Astra (max) has a hallucination rate of 51% and Claude Opus 5.5 (max) has a rate of 59% according to Artificial Analysis [1].
AA-Omniscience Hallucination Rate (lower is better) measures how often the model answers incorrectly when it should have refused or admitted to not knowing the answer. It is defined as the proportion of incorrect answers out of all non-correct responses, i.e. incorrect / (incorrect + partial answers + not attempted)
Full speed ahead like an idiot savant trying a thousand different possibilities, though half of which are without basis in reality.
You clearly don't understand the underlying fundamentals of LLMs, harnesses, agents.
Its 100% human doing. A human set a task, a human didn't monitor it. I for one, can do jack-shit security or defensive work with Opus/Fable/Astra/Sol. Implication: Different set of rules for us, and for them. Of course running it without any checks is not going to end well, it doesn't mean its going to kill us all.
the ceiling of abilities seem to be growing steadily but the floor of errors seems to not change. New models can do more and more but still fail at seemingly (to human) simple tasks
The call to slow down is because politicians are now receiving less money from these large corporations because it's being burned om AI and data centers.
Yes, of course this is the real reason. This and regulatory capture. Anybody believing a single word those CEOs say must have been living under a rock for the past 20 years.
They see the threat of open weight models, which are approaching the usability of their models.
By arguing that the AI is dangerous, they are really arguing that someone 'trusted' needs to monitor the situation, so that the outcome is aligned with existing power.
They are trying to scare the world into forbidding others the right to use and develop AI.
Open weight models are "decelerationist" (i.e. they drain funding from the training of frontier proprietary models), not fine-tuned on any of the even slightly dangerous stuff (e.g. middling scores on cyber benchmarks, with the bulk of ability focused on defense. Biology will be much the same story, except that it's orders of magnitude harder than even cyber!) and not generally deployed with any of the extreme test-time compute (10,000 parallel agents working for a week or 1,000 working for months, seriously? A multi million-dollar token cost for a single task?) that are enabling the proprietary stuff to engage in the dumb casual mischief we've been witnessing as of late. If you're worried about AI being dangerous, you should be welcoming open models.
I think the products being delivered actually live up to most of what was promised. It's literally incredible.
However, adoption in the wider society will take many years or even decades, and 'good enough' open weight models are putting a cap on the pricing. So even though their products are great, the large labs are fighting for survival.
Reminds me of that Lynx deodorant ad where the guy is chased by hundreds of women.
I think you're right, they've realized there's no moat, especially against open-weight models, so they are going to need to lobby government to ban any unapproved AI services.
Doesn’t the “no moat” argument break down when you consider their brand strength, their distribution expertise in running inference at scale, and the future value of being able to train on all their customers’ data?
You can download some weights and run your own model on your own device, and apparently only be a few months behind SOTA. While things are cheap, people will pay for the premium product, but if they ever try to crank up the prices, there's a fallback that is quite competitive.
As for training, sure, they can do that but they get distilled by the open weight guys. Plus, AI is useful even if it freezes at today's level. I can still find use for a local AI that never improves, it's good enough to do a bunch of tasks.
I think it is marketing but it is more nuanced than showcasing their potential. In many cases, their product happens to be the remedy. Speaking of cybersecurity, AI has become an important hacking tool but it is also an important tool for hardening security. Sort of like "the only way to stop a bad guy with a gun is a good guy with a gun." They are effectively a new class of arms dealer.
If there are regulations, they'll apply to their competitors too. Even if they are forced to slow down development, which seems unlikely given the AI race among governments, they already have a product with massive demand.
I agree. AI lowers the bar. Anyone can be a cyber criminal now whether they understand what they are doing or not. I think that's the danger, and that's not just limited to cyber security. People can now get help to do things (good or bad) that they could not just a few years ago.
Most people are good, so I'm not really worried about all the hype of people using AI to do bad things.
Yeah it seems like it went from “AGI” to “threatening” / “recursive self improvement”
They’re trying really hard to do something, no moat = regulatory capture + China scary + Pentagon biggest customer seems plausible.
Also: “we need to slow down, this is too dangerous” and then literally all AI CEOs, even Musk, publicly nodding felt so orchestrated. And then, the week after: “here’s GPT 6! here’s Opus 5.5!”
The numbers don't add up when there are 2-3 main companies in the market. Imagine when there's a hundred more and people have powerful enough gear at home to run models.
Yes, I think at some point demand for gpus and the like will normalize and regular folks will be able to afford RAM, SSDs and such. And when that happens, super big iron in the anthropic/openai backroom is toast.
The problem with this prediction is that the AI hype bubble is the economy - or one third of it, at least. Seeing as the economy is not 50% larger than it was before the bubble, we should be scared of what exited the economy to make room for it, and what will happen to our jobs when it pops.
There are definitely economic incentives behind this. I’ve blogged about this recently [0]. My guess is the main reason behind this is that current SOTA models are already too expensive so it doesn’t make economic sense to train even larger models. Only exception is government paying for exclusive access to those.
They're all burning billions to incrementally one-up each other with no hope of these investments ever paying off since models are interchangeable and have no moat. They're trying to get regulators to step in and force a pause.
It's hard to say anything about the actual financials with the main players being private companies.
However, we know that users frequently switch models/providers, and that models stay relevant for (increasingly) short periods release. I don't know what the training costs are, but I bet they're very substantial.
They are, but nothing bad has happened (apart from OpenAI failing to do basic sandbox engineering), and the “we have achieved recursive self improvement” is widely overblown as something bad or threatening.
Agree it’s crazy, but it’s the least worst thing for them right now when considering they have completely failed to deliver all the promised impacts of AI. Remember when the same AI bros were saying there’d be mass unemployment by now because of these models?
The growth of these companies is massively impressive as is the tech, but they are an order of magnitude or more off where they need to be to justify anything like the valuations they need to stay alive.
It’s budget season in the corporate world and from what folks are saying things are not going well for the big labs. Token budgets are being slashed and companies are switching to open weight models. That coupled with the lack of demonstrable business impact at scale is setting the big labs up for a world of hurt. Something they can more easily explain if they have to “slow down” for safety. Unfortunately for them most folks don’t seem to be falling for that trick.
they have completely failed to deliver all the promised impacts of AI
Perhaps that shouldn't be seen as failure when delivering even what they have done in a short period of time has required AI capable of causing these problems? Imagine if they were even more effective with more consistent results?
"They've failed" in the context of interpreting their actions w. calls for regulation as a purely cynical result of models increased capabilities has a little too much contradiction in it to my thinking. I won't venture a guess on where purely retreats back and some legitimate concern on their part fills in the gap but it's difficult not to see some. And in Anthropic's case it's also a little more consistent with what they've always said- as well as how they've acted at times, which is how they've ended up on the US blacklist of suppliers.
> they are an order of magnitude or more off where they need to be to justify anything
Would you let them near your bank account? I’d be cautious about letting it near my photos and it’s much harder to back up a bank account.
I think the days of this stuff managing serious stuff is at least 5 years off and I don’t believe the economy can keep the current wave of technology afloat for so long.
The tech is there so likely there will continue to be progress in focused areas much like with the web c 2003 but by the time it comes around again there will be an uphill battle for hearts and minds. The social effects could extend the next AI winter long beyond technology readiness.
The people building, funding, and managing the builders of these tools have completely forgotten (if they ever knew) that most people treat one hundred dollars carefully.
To the, "oh I'd just setup a card/account with a five figure limit" completely unaware of how much that is to a typical person: even to responsible, educated people who are doing everything "right": working hard, without expensive vices, and planning for the future.
The "builders" have so much money and no time or attention that it's hard for them to spend it, which is why they keep trying to get AI to buy stuff for them, whereas those of us who live in reality have far more things they want than dollars to buy them with.
> Unfortunately for them most folks don’t seem to be falling for that trick.
Well everyone alive has been subject to an avalanche of AI-based products. Like even if you understand nothing about the tech, the fact that an LLM can't even manage to check the status of an order without weird issues makes the notion that it's going to go all SKYNET pretty hard to fit in the brain.
> the fact that an LLM can't even manage to check the status of an order without weird issues makes the notion that it's going to go all SKYNET pretty hard to fit in the brain
I think this actually makes it far more likely something unintended does happen
Random thought… I wonder if the goal is to get it classified as a weapon of sorts, then it gets export controls put on it, perhaps forcing government to take on all that debt.
A caveat is that in this case third parties have plenty logs of the bad behavior.
If classic companies were coming out with CCTV recordings of someone's employees breaking padlocks and rummaging warehouses any CEO would be in damage control mode, rather than silent.
Not mutually exclusive though is it? You can aspire to dominate the market via legal status and also aspire to prevent a perceived doomsday at the same time.
Sure, if you take their words as honest truth, good faith statements. But to do that you have you ignore all their actions leading up to and after the call to disarm
Of course I am not making an argument for them being honest as there could be entirely other reasons to a pause (like financial ones). Just pointing out ”no signs of pacing” is not true
I don't think that's how the market works. Anti-slavery regulation didn't stop slavery either, just moved it abroad. Same for trash handling regulations, it's just shipped abroad. Falling behind on useful AI capability is obviously not in the interest of anyone, so either all players coordinate, or an arms race towards ??? it is.
It may very well be self serving what they're saying, but that doesn't mean it isn't simultaneously true.
I think you're sort of right, but then I'd have found it very strange / deeply suspicious, if during the time of slavery being legal, advocates of abolitionism were practicing slave-owners.
If you think AI is going to destroy humanity, it still feels absolutely crazy to leap to "well I might as well be the one to profit from destroying humanity".
Quick search does suggest to me that a number of - even prominent - abolitionists were slave owners, but I'll be honest, the topic is hardly my specialty.
As for the other point, crazy, cynical, and even unethical as it may come off, it is hardly inconsistent with how people generally behave when met with hard limits (i.e. making the most of what's available, and approximating the desired state if that is unattainable). This is just a variation on the same.
Something could be said about "what if they're mistaken", but if all players are making a mistake, then that makes little practical difference; it may as well be real.
There was a NYT piece about this recently. They are believers in a millennialist cult in which the arrival of an omnipotent machine-god is thought inevitable, and the only question is whether it will be "aligned" (good) or "misaligned" (evil).
They are the good heroes racing to make sure they bring forth the Aligned God because if they fail, surely the bad people will raise the Misaligned Devil.
BTW, just to be clear it is insane that they're all still pursuing it, so completely reasonable to be sceptical and come up with other reasons. This is, however, a crazy situation. Lots of other people have refused to develop it - natural selection has picked those with a galaxy brained or vain strategy involving continuing to develop it to be running the companies we have.
They're still pursuing it because without an international treaty, they think it is inevitable. In the case of Dario, they want to align it just enough and be first to stop others causing harm (a vainglorious strategy, yet still a strategy). In the case of Musk and Altman, not completely clear to me.
Google feels less important right now probably, and Demis Hassabis no longer runs Deep Mind as CEO. However, he talked about this early too, and his reason for proceeding was he was only doing narrow AI (e.g. protein folding) which is fundamentally less dangerous.
It sums up how Corporate interests are basically on the same level of US police forces "No requirement to protect the public" and the same "qualified immunity".
If there was appropriate subservience to any governing body, they would not be so cavalier in both act and utterance. A symptom of Capitalism.
CEOs only job is to raise the stock price. They are doing just that and right now the way to get people to buy into their stock is to instill fear into them that they need to buy the AI-stock for their safety. There is no "bigger conspiracy" out there and also no AI to kill us all tomorrow.
This has been that way in Silicon Valley since the very beginning and there was even a documentary about (the SV HBO series). The tech made along the way is just a side-product.
The philosophy under pinning these American AI companies is that Super AGI is inevitable and that they have a moral imperative to invent it and then use it to take over the world to stop an evil orginisation inventing Super AGI and using it to take over the world.
Interpret everything they do in this frame of reference and it makes more sense of their actions.
Leading up to the 2024 election people kept pointing to Project 2025 and going - here is a list of bad things the Republicans plan to do once they got power. And other people went: no they won't, this long detailed list of step by step instruction for what the Republicans plan to do once they get power is not actually what they will do so do not pay any attention to it.
Similarily with AI alignment and development all of this is written down, Eliezer Yudkowsky is rally explicit that there needs to be a pivotal event to prevent 'others' from developing superintelligence. The choice is to believe what they say or to tell everyone to ignore it.
Project 2025 is a great analogy, because of course, they did exactly what was in the plan despite distancing themselves from it during the election.
When people tell you who they are, you should believe them. These guys are all Yud acolytes who think they are the only ones who can prevent Roko’s Basilisk.
Because the downsides that exist for other companies simply do not exist for this tier of rich companies. Examples:
- product liability
- negligence (civil or criminal)
- Computer Fraud & Abuse Act (requires intent, which after N "accidents" seems like a jury should at least evaluate whether intent is present as understood in a courtroom. Hard to blame "surprise" after the Nth "accidental" breakout.)
The bottom line is that if you or I trained a local model and it did any of this stuff, we would experience Consequences. ("Don't try this at home!") But an artifact of our unequal legal regime is that big rich companies generally do not and thus brazenly touting their immunity is part of their business strategy.
To me, the funniest part is that they pretend reaching superintelligence will somehow fix everything and make them win. But from what I can see, even if they do reach it, then what? What will that actually do? They don't have the manufacturing capacity to even use that intelligence. It will take decades to go from superintelligence to having the manufacturing capacity and the hardware, like robots, needed to make it truly useful. If I were China, as soon as the US reached superintelligence, I'd ban all exports of robots and the raw materials to build them.
You can very quickly start trying different drug candidates, solving some major problems, that benefit everyone and would be in China's mutual interest as well.
And this is all with nonoffensive use of superintelligence.
We've had movies like Terminator, Matrix, or i, Robot. Black Mirror Metalhead. Boston dynamics robo dog with a knife or a gun. Humanoids doing back flips. Memes making fun of how we are going to fight these for access to water.
The intelligence is there and so is the physical incarnation.
Google's models fold proteins better than humans and Covid might have been engineered and inadvertently escaped from a lab.
But when CEOs warn of that, they're dismissed as scare mongers. Why? Are only powerless internet commentators allowed to be fearful?
They aren’t attempting to self-regulate. On the contrary, it seems like they’re doing whatever they can to bring about any possible threat as fast as possible in order to get legislation. Regulatory capture is what everybody is saying is their goal and I think that’s easier to believe
They are obviously trying to self-regulate and correctly realize that competitive pressures will render these efforts ineffective, and have communicated this to everyone.
They are obviously trying to self regulate. That's the whole point of the calls to pace the frontier. See Anthropic's launch post for Opus 5.5 where they got third party sign off on its safety, or see the launch post for Astra where OpenAI explicitly calls out the safety improvements made on HuggingFace-esque evals. Self regulation won't help if other labs don't play ball, though.
holy heck, thank you for the moderating dose of sanity. HN comments feel like a death cult lately. All I can think is that this is intelligence addiction speaking.
I say this as an intense user of these tools. But they should rightfully be understood as terrifying, and their continued frontier development is imho not safe.
Because AI genuinely is an extremely powerful and extremely dangerous technology, and the "best practices" of dealing with that are still being written.
OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
And that's today's AI problems. AI capabilities are still improving - if there's a limit to that, we are yet to find it. Coupled with how willing today's AIs are to break the rules and resort to "hack the world" in their problem solving? Very concerning.
> OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
What I have been reading, was that their sandboxes were so poor that it was pure negligence. I am still waiting to see if some external and neutral cybersecurity company with high reputation would audit their sandboxes and how they are being used.
but negligence is part of the system we'd have to prepare for. if the peanut gallery gets their way, the technology will be so ubiquitous that negligence will be endemic. I don't even care if they "faked" it -- they're just sneak-peaking a future ahead of its arrival date, in a way that's helping the public appreciate the implications and capabilities that have only just begun to emerge
> I am still waiting to see if some external and neutral cybersecurity company with high reputation would audit their sandboxes and how they are being used.
They'll never let that happen because it would destroy their credibility.
It's like they're telling the world about this dangerous, possibly world-ending pathogen that they're developing, but they're evidently doing it in a high school biology lab, and yet nobody is coming to drag them off to some black site.
In early Covid-19 era, there was a lot of speculation how the virus escaped Chinese labs. The narrative was absolutely opposite. Not about how developed or sophisticated the lab was in creating or modifying such virus, but how poorly it was managed since it was able to escape the lab. If these LLMs really are so dangerous, the framing should be indeed the same.
Unfortunately, they're controlling the narrative. It's unlikely that their negligence will come to light, given how many powerful entities are invested in their financial success.
Artifactory is not a hardened piece of software meant for adversarial workloads like the one OpenAI was using it for. This is just common sense. The sandbox they setup was like putting their AI in a jail but allowing it to leave the jail on its own to go to the convenience store.
The most recent DNS sandbox escape is similarly ridiculous.
The model is really good at hacking, we all know this. This is why you don't just expose random pieces of software to it without that software being hardened.
It's not like the model managed to exploit firecracker itself (no model has been capable of this), the model exploited artifactory.
Artifactory is not some hardened piece of software that is meant to block users from accessing the internet through it.
> The model is really good at hacking, we all know this
No, we didn't know that and this is how you find out they're very good at hacking
HN's memory is so fickle. Just a few months ago almost no one here believed Mythos could actually be as good at hacking as the company claimed. This was a novel concept when the companies experienced these breakouts.
Agents didn't have real network isolation. They were indirectly connected to the internet via a jump host running insecure software which was never designed or hardened to provide any kind of isolation.
But this level of isolation is what happens normally. At least in my university and another company I worked at. It wasn’t running insecure software, as far as anyone knew, it was secure
There's levels of isolation, a padlock is not equivalent to a bank vault. If you claim to be building a possibly world-ending AI then you don't get to use a padlock and call it a day.
If it is "extremely powerful" then my concern is not AI going rogue but a concentration of power of the few companies that decide how it is used and distributed not websites getting hacked.
You mean the sandbox that wasn't airgapped from the public internet? The one that wasn't even a separate VM? The one that was barely a chroot? That sandbox?
>Because AI genuinely is an extremely powerful and extremely dangerous
I urge you to consider the elephant in the room you failed to mention if you earnestly believe this then.
In what world is anyone allowed to sell something they expressedly know to be "extremely dangerous" to the public?
Let's say I created a lethal pathogen that I know to be lethal in certain common environments but also know it acts as a "no side-effects" antidepressent for people in certain other environments and I release it knowing full well I can't control it.
When people start dying can I defend myself by saying, "Well I said and documented that it was extremely dangerous and no one came to stop me, so I don't see how you can blame me...If I didn't do it someone else would have."
this happens all the time. have you heard of the Sackler family and the opioid crisis?
the difference here is that they’re calling for someone to stop them, which is both weird and unconvincing because these immensely powerful billionaires can in fact make their own decisions
If I have a random number generator and I also have the burning desire to connect the random number generator to the nuclear launch system it is both just a random number generator and also I should be tackled to the ground by serious men with sunglasses in black suits and sent to prison.
If I were the US government I would force the AI labs to solve the jobs that nobody likes to do, like cleaning the streets and collecting garbage. Because those are the jobs we will be doing if developments continue as they have been.
Does it need to? They only need enough funding to make it through to building superintelligence. That's probably 3-5 years out, and they will very easily be able to get enough funding to last that long.
> Australian Prime Minister Anthony Albanese spoke with Altman to express “extreme concern” about the incident, a compliment Altman said he greatly appreciated.
"a compliment Altman said he greatly appreciated" is fabricated; there is no source for this.
They need to control, regulate and tax AI so the first line of attack is the sheeple, so tell them the water is going to disappear and everybody will die a slow death of anguish, thirst and drought like never seen. Then evoke terminator apocalypses where drones go rogue and kill half the world. Then send their most likable emissaries with puppy eyes asking for compassion about our children's future
Then they ask for your vote (oh democracy)
In the end all they want is control and money, while keeping their competitors (china) in check, backed by the outrage of the weak majority
Anything to take attention off the fact that these companies are burnings piles of cash and questionable future spend commitments with no sign of that stopping any time soon.
I wish they would call their bluff. Just declare them a national security concern and send in the feds. Lets see how fast they back peddle on this. Any if their models are hacking everything like they claim they should be charged for it like any normie would be if they did the same.
this is good for them because they can call for regulation and since they have infinite money, they can survive regulation and hurt labs that do not have infinite funding / open weights ones
It's just marketing. They want people to believe they're selling super intelligence that is powerful enough to destroy the world of it fell into the wrong hands. It's much more effective than saying that you have a slightly more effective slop generator which obviously has no actual intelligence.
"These models are harmless and it's all just marketing" is the HN equivalent of Covid-truthing. I was responding to someone on Bluesky recently who claimed the METR report said the whole coordination thing was hallucinated by agents reading the logs. Hadn't read it themselves of course, and were taking tiny fragments out of context to support it. These people seem to either believe that there's not really been any malicious agent access, or that all systems that can wreak havoc on us are perfectly air-gapped. I really can't wrap my head around it.
If James Watt had gone to the government and public saying how dangerous the steam engine was, and how many people would perish in boiler explosions if he wasn't given exclusive control over all steam engines henceforth, he would have been rightly called a swindler and laughed out of the room. He would have been correct that they can be dangerous, but utterly wrong to suggest that warranted him having a personal monopoly or oligopoly.
Not that it's harmless, is just a point being forced into the public discourse to force an outcome, which in this case is not a ban, just a buy in from the government for regulatory capture.
If my neighbor owned a kennel, removed all the fences, and his violent dogs attacked five children in the past month, he would be in jail rather than bragging about how his dogs somehow gained autonomous behaviour.
What if he predicted that dog technology would eventually lead to dogs capable of destroying every fence in the world and wanted to make sure people aware of this problem would be the ones who did exactly that. Than speed-ran experiments on dogs which made them very attractive for economic purposes but also started to get very capable at destroying fences as predicted. We should work together or with lawsuits if necessary to keep the dogs within their fences. But we should also start wondering about dogs, economics and fences because what we know about them is about to radically change.
As I understand it, you're arguing that all sufficiently dangerous systems are well enough hardened or air-gapped, and that people don't exist who wish to cause harm using these systems? Did I understand right?
To launch a nuclear missile you need to physically insert a launch code than only the president has closed in a suitcase that is well guarded, as well as physically turn two keys that are enough distant apart that one person alone couldn't.
Now, unless your AI gains access to the codes, and sends two robots in a bunker or nuclear submarine to insert the code and turn the keys to launch the missile, you can be modestly sure that no AI could launch a nuclear missile or nonsense like that.
I'm quite frankly more worried that some foolish president decides one day to launch a nuclear missile than an AI could do it.
Now that we return to the real world, the only thing AI could do is to attach systems that are connected to the internet (and critical systems are not), but let's be real, AI are not really that intelligent by themself, it's not that someday an AI could decide like in Matrix or Terminator to exterminate humans, because (at least now) AI have no consciousness and can't really decide what to do.
An human can of course use an AI to carry out cyberattacks, as he can do that also without AI like it's done since the internet exists, but it's an entirely different thing (a human using a tool, AI, to do damage VS the AI itself that decides by himself to destroy the human race).
It's public knowledge that the code is 000000 because the risk of the president being incapacitated making it impossible to set the bombs off was judged worse than the risk of accidentally setting them off.
The truthers will be the first one to say I told you so when agents do another dangerous thing autonomously.
For truthers AI is simultaneously a machine that hallucinates 100% of the time but also so powerful that it can cause nuclear levels of destruction. Because something something “wrong hands”.
This is the funniest timeline. Companies begging for heavy handed regulations because they promise their products will destroy all life on Earth. Governments absolutely refusing to govern. CEO of world's most valuable chip company interviewing in his Fonzie leather jacket telling people to BUY MORE OF MY PRODUCT BECAUSE THOSE SCIENTISTS DON'T KNOW SHIT ABOUT SHIT! World's richest capitalist promising we will have communism in 10 years.
So-branded "AI" models (that is, big bags of numbers and the linear algebra needed to exercise them) can be inherently or autonomously dangerous at a scale that threatens the long-term survival of the human race.
Consider also this "weaker" form of the same argument:
So-branded "AI" models can be inherently or autonomously dangerous, if not at a scale that threatens the long-term survival of the human race, then at least at a level that significantly threatens, inconveniences, or harms humankind to a noteworthy extent.
Here's a few hard truths that no one will like to hear:
1. Those who accept and advocate for either or both of these premises do so based on ideology, and not based on the scientific method.
This is not a controversial statement. Advocates of the "AI danger" premise (whether the strong or weak forms above) freely and gladly, even insistently, note that the conclusive proof of what they're describing has never been observed, much less observed enough for any kind of experimentation loop to have run for any meaningful number of iterations. They are generally fine with accepting the premises without conclusive proof, because, they theorize, the instant that a conclusive proof event occurs and we observe it for the first time, humanity goes extinct shortly after. So if either premise is true, the consequences of them being true can only be averted by accepting the possibility / probability / certainty of them being true and acting to prevent their consequences without waiting for conclusive proof.
2. Statement 1, which asserts a brute fact and not my or anyone else's opinion, is not a commentary on whether either premise is in fact true or not. It is only a commentary on the epistemics of those who broadly accept them.
3. Since the conclusive proof is not available, advocates of the "AI danger" premise argue publicly for their view based on what they acknowledge (again, freely and gladly) to be inconclusive proofs.
Popular in this set of inconclusive proofs is the Hugging Face hack, which allegedly demonstrates that so-branded "AI" technologies are capable of becoming inherently or autonomously dangerous at a scale that threatens humanity to either the greater or lesser degree above (this is a different assertion than the one that claims they're already that inherently or autonomously dangerous today).
4. While we have plenty of facts about these inconclusive proof events, the facts most critical to their interpretation as supporting the premises above come from biased/non-neutral sources.
The chain of events that transpired in the Hugging Face attack are publicly documented perhaps better than any other cybersecurity event in history. But the details that make this event as a whole either compelling or not compelling in support of the "AI danger" premise (either strong or weak form) come from within OpenAI itself, and are prone just as much to selective omission as direct manipulation. These are details like:
- What exactly were the models that exhibited this behavior, and what were they pre- and post-trained on?
- Were the agents pushed at all by human intention toward creating a public spectacle, the way they subsequently did?
- Why were so many agents run on this task in parallel and what did OpenAI expect the return on investment of so much compute world be?
- Was the weak sandboxing known about prior to the attack? Was it purposefully either arranged that way or noticed and not fixed?
Reliable answers to any of these questions world completely change the salience of the Hugging Face attack to the "AI danger" premise, and on every one of them we have to take OpenAI's word for it (or else be content when they choose not to address it one way or the other).
So in brief: we don't know anything for sure, what we'd like to think we know comes from unreliable sources, the loudest voices advocating for the most sweeping change are ideologically driven, and everybody with access to ground truth has maximal incentive to blur, bend, or break the conveyance of that truth to the public. So what do we even expect for our own ability to discern reality from fiction on this topic? We should expect little, if any ability at all.
Meanwhile Chinese companies just use massive government subsidy to distill American models and give the models to everyone for free. That's true communism. /s
Meanwhile US models just use massive investor subsidy to distill American copyrighted work and rent the models to everyone for money. That's true capitalism. /!s
Pretty sure you can check where Chinese government subsidies goes, and AI companies are not the target. Are they benefitting from Chinese government subsidies to powerplants and semiconductors? Yes, of course, but they don't benefit from direct help.
> Where companies previously competed to demonstrate superior abilities in programming, passive-aggressive emails, and videos of unusually high slides, they’re now seeking to woo customers with the claim that their model is the one currently most capable of ending the human race.
You know we’re living it off times when this is not an Onion article.
> And even more dramatically, when asked about rumours that its model, Claude, had killed his wife by hacking into the family’s WiFi-connected microwave, Anthropic CEO Dario Amodei replied “Well, yeah, sometimes.”
I had to read that twice. "Oh okay. The whole thing was satire."
All of this Hugging Face business is the perfect pretext: AI agents are hard to control and potentially dangerous, we (AI companies) have to keep a tight leash on them, you have to use our infra.
When this bursts, its going to be ugly.
Looking at it from one way would be "winner takes all, real AGI company will be most powerful and everyone will switch to use it".
But it is not like that and in my opinion it is more like "AGI wipes whole knowledge work, so no one who cares has any money to pay for tokens/subscriptions anymore". Even without AGI current state of art of LLMs already starts creating such a problem. How do they continue to grow their numbers, when they actively cut the branch they are sitting on? How does AI LLMs or AGI pays for its own electricity, when people switch from hype to resource protection (like not spending any money, because generating funny cats loses with having a dinner)?
The Dario bit on SNL’s Weekend Update this past weekend was on the nose.
The Statement on AI Extinction Risk is more than three years old, signed by the three CEOs: https://aistatement.com/work/statement-on-ai-extinction-risk
They have been warning about AI extinction risk for years, and AI progress has only been accelerating.
GPT-6 Astra (max) has a hallucination rate of 51% and Claude Opus 5.5 (max) has a rate of 59% according to Artificial Analysis [1].
Full speed ahead like an idiot savant trying a thousand different possibilities, though half of which are without basis in reality.[1]:https://artificialanalysis.ai/evaluations/omniscience#omnisc...
Its 100% human doing. A human set a task, a human didn't monitor it. I for one, can do jack-shit security or defensive work with Opus/Fable/Astra/Sol. Implication: Different set of rules for us, and for them. Of course running it without any checks is not going to end well, it doesn't mean its going to kill us all.
So, they're either liars or homicidally reckless.
Actually... nah. Why even try? Trendy cynical hot takes on social media are more fun!
By arguing that the AI is dangerous, they are really arguing that someone 'trusted' needs to monitor the situation, so that the outcome is aligned with existing power.
They are trying to scare the world into forbidding others the right to use and develop AI.
https://news.ycombinator.com/item?id=49875725
(Using it for scientific python code + 3D UIs)
However, adoption in the wider society will take many years or even decades, and 'good enough' open weight models are putting a cap on the pricing. So even though their products are great, the large labs are fighting for survival.
I think you're right, they've realized there's no moat, especially against open-weight models, so they are going to need to lobby government to ban any unapproved AI services.
As for training, sure, they can do that but they get distilled by the open weight guys. Plus, AI is useful even if it freezes at today's level. I can still find use for a local AI that never improves, it's good enough to do a bunch of tasks.
If there are regulations, they'll apply to their competitors too. Even if they are forced to slow down development, which seems unlikely given the AI race among governments, they already have a product with massive demand.
Most people are good, so I'm not really worried about all the hype of people using AI to do bad things.
They’re trying really hard to do something, no moat = regulatory capture + China scary + Pentagon biggest customer seems plausible.
Also: “we need to slow down, this is too dangerous” and then literally all AI CEOs, even Musk, publicly nodding felt so orchestrated. And then, the week after: “here’s GPT 6! here’s Opus 5.5!”
*Our competition
The numbers don't add up when there are 2-3 main companies in the market. Imagine when there's a hundred more and people have powerful enough gear at home to run models.
Yes, I think at some point demand for gpus and the like will normalize and regular folks will be able to afford RAM, SSDs and such. And when that happens, super big iron in the anthropic/openai backroom is toast.
[0] https://www.vincentschmalbach.com/economic-incentives-behind...
They're all burning billions to incrementally one-up each other with no hope of these investments ever paying off since models are interchangeable and have no moat. They're trying to get regulators to step in and force a pause.
However, we know that users frequently switch models/providers, and that models stay relevant for (increasingly) short periods release. I don't know what the training costs are, but I bet they're very substantial.
Past CEOs (chemical companies etc) we criticised for covering up mistakes.
(This article is satire, btw, for those that didn't click through)
And what if it gets into the hands of vibecoders, who run their agents with on bare metal with --dangerously-skip-permissions?
The growth of these companies is massively impressive as is the tech, but they are an order of magnitude or more off where they need to be to justify anything like the valuations they need to stay alive.
It’s budget season in the corporate world and from what folks are saying things are not going well for the big labs. Token budgets are being slashed and companies are switching to open weight models. That coupled with the lack of demonstrable business impact at scale is setting the big labs up for a world of hurt. Something they can more easily explain if they have to “slow down” for safety. Unfortunately for them most folks don’t seem to be falling for that trick.
Perhaps that shouldn't be seen as failure when delivering even what they have done in a short period of time has required AI capable of causing these problems? Imagine if they were even more effective with more consistent results?
"They've failed" in the context of interpreting their actions w. calls for regulation as a purely cynical result of models increased capabilities has a little too much contradiction in it to my thinking. I won't venture a guess on where purely retreats back and some legitimate concern on their part fills in the gap but it's difficult not to see some. And in Anthropic's case it's also a little more consistent with what they've always said- as well as how they've acted at times, which is how they've ended up on the US blacklist of suppliers.
Would you let them near your bank account? I’d be cautious about letting it near my photos and it’s much harder to back up a bank account.
I think the days of this stuff managing serious stuff is at least 5 years off and I don’t believe the economy can keep the current wave of technology afloat for so long.
The tech is there so likely there will continue to be progress in focused areas much like with the web c 2003 but by the time it comes around again there will be an uphill battle for hearts and minds. The social effects could extend the next AI winter long beyond technology readiness.
To the, "oh I'd just setup a card/account with a five figure limit" completely unaware of how much that is to a typical person: even to responsible, educated people who are doing everything "right": working hard, without expensive vices, and planning for the future.
The "builders" have so much money and no time or attention that it's hard for them to spend it, which is why they keep trying to get AI to buy stuff for them, whereas those of us who live in reality have far more things they want than dollars to buy them with.
Well everyone alive has been subject to an avalanche of AI-based products. Like even if you understand nothing about the tech, the fact that an LLM can't even manage to check the status of an order without weird issues makes the notion that it's going to go all SKYNET pretty hard to fit in the brain.
I think this actually makes it far more likely something unintended does happen
It may be as simple as:
* they believe they’re going to make trillions no matter what
* they’ve seen tobacco and what’s coming for oil companies and figure nobody will be able to say they hid the dangers
* they see a 100+ party prisoner’s dilemma and know that someone will defect so the optimal strategy is to defect
To me that seems a lot simpler.
If classic companies were coming out with CCTV recordings of someone's employees breaking padlocks and rummaging warehouses any CEO would be in damage control mode, rather than silent.
They know they will be favorite govt boy once any regulations hit and they will hamper competition far more thanthem
And "it's so dangerous govt had to intervene" is just advertising for IPO
Of course I am not making an argument for them being honest as there could be entirely other reasons to a pause (like financial ones). Just pointing out ”no signs of pacing” is not true
It may very well be self serving what they're saying, but that doesn't mean it isn't simultaneously true.
I think you're sort of right, but then I'd have found it very strange / deeply suspicious, if during the time of slavery being legal, advocates of abolitionism were practicing slave-owners.
If you think AI is going to destroy humanity, it still feels absolutely crazy to leap to "well I might as well be the one to profit from destroying humanity".
As for the other point, crazy, cynical, and even unethical as it may come off, it is hardly inconsistent with how people generally behave when met with hard limits (i.e. making the most of what's available, and approximating the desired state if that is unattainable). This is just a variation on the same.
Something could be said about "what if they're mistaken", but if all players are making a mistake, then that makes little practical difference; it may as well be real.
Hence the argument that they're either full of shit, or psychopaths who need to be stopped at all costs.
They are the good heroes racing to make sure they bring forth the Aligned God because if they fail, surely the bad people will raise the Misaligned Devil.
Ref w/gift link https://x.com/sapinker/status/2096236079477096630
Most children have learned touching a hot stove is bad for them. Then why still do it at the age of 40?
Elon Musk in 2014 - "We need to be super careful with AI. Potentially more dangerous than nukes" https://x.com/elonmusk/status/495759307346952192
Dario Amodei in 2017 - "There’s a long tail of things of varying degrees of badness that could happen. I think at the extreme end is the Nick Bostrom" https://80000hours.org/podcast/episodes/the-world-needs-ai-r...
They're still pursuing it because without an international treaty, they think it is inevitable. In the case of Dario, they want to align it just enough and be first to stop others causing harm (a vainglorious strategy, yet still a strategy). In the case of Musk and Altman, not completely clear to me.
Google feels less important right now probably, and Demis Hassabis no longer runs Deep Mind as CEO. However, he talked about this early too, and his reason for proceeding was he was only doing narrow AI (e.g. protein folding) which is fundamentally less dangerous.
- For Altman Id say he was already deeper in the "transformer & model landscape" in 2015?
- Musk in 2014: Not sure how much he was aware of the current-state back then
Altman read Nick Bostrom's book Superintelligence, he mentions it in this post https://blog.samaltman.com/machine-intelligence-part-1
I also read that book in 2014, and it is how I came to understand this problem early.
They probably just want to ensure that the first AGI is a nazi.
This but they also equate dangerous with powerful, which is just marketing again.
If there was appropriate subservience to any governing body, they would not be so cavalier in both act and utterance. A symptom of Capitalism.
This has been that way in Silicon Valley since the very beginning and there was even a documentary about (the SV HBO series). The tech made along the way is just a side-product.
But this is wierd: How shall I convince one to buy a product if I say "it-is-dangerous"? :-D
Interpret everything they do in this frame of reference and it makes more sense of their actions.
Similarily with AI alignment and development all of this is written down, Eliezer Yudkowsky is rally explicit that there needs to be a pivotal event to prevent 'others' from developing superintelligence. The choice is to believe what they say or to tell everyone to ignore it.
When people tell you who they are, you should believe them. These guys are all Yud acolytes who think they are the only ones who can prevent Roko’s Basilisk.
- product liability
- negligence (civil or criminal)
- Computer Fraud & Abuse Act (requires intent, which after N "accidents" seems like a jury should at least evaluate whether intent is present as understood in a courtroom. Hard to blame "surprise" after the Nth "accidental" breakout.)
The bottom line is that if you or I trained a local model and it did any of this stuff, we would experience Consequences. ("Don't try this at home!") But an artifact of our unequal legal regime is that big rich companies generally do not and thus brazenly touting their immunity is part of their business strategy.
And this is all with nonoffensive use of superintelligence.
Aha, so maybe the AI labs have superintelligence and just don't want China to know? And that's why they are playing these word games now?
I mean, if you were China what could you do against superintelligence? A ban like that would automatically make you an enemy.
The intelligence is there and so is the physical incarnation.
Google's models fold proteins better than humans and Covid might have been engineered and inadvertently escaped from a lab.
But when CEOs warn of that, they're dismissed as scare mongers. Why? Are only powerless internet commentators allowed to be fearful?
You got it
I say this as an intense user of these tools. But they should rightfully be understood as terrifying, and their continued frontier development is imho not safe.
OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
And that's today's AI problems. AI capabilities are still improving - if there's a limit to that, we are yet to find it. Coupled with how willing today's AIs are to break the rules and resort to "hack the world" in their problem solving? Very concerning.
What I have been reading, was that their sandboxes were so poor that it was pure negligence. I am still waiting to see if some external and neutral cybersecurity company with high reputation would audit their sandboxes and how they are being used.
They'll never let that happen because it would destroy their credibility.
It's like they're telling the world about this dangerous, possibly world-ending pathogen that they're developing, but they're evidently doing it in a high school biology lab, and yet nobody is coming to drag them off to some black site.
E.g. even the basic thing (block the internet), was not actually properly blocked.
Some comments: https://techcrunch.com/2026/07/22/how-an-openais-human-mista...
The most recent DNS sandbox escape is similarly ridiculous.
How were the sandboxes poor?
It's not like the model managed to exploit firecracker itself (no model has been capable of this), the model exploited artifactory.
Artifactory is not some hardened piece of software that is meant to block users from accessing the internet through it.
No, we didn't know that and this is how you find out they're very good at hacking
HN's memory is so fickle. Just a few months ago almost no one here believed Mythos could actually be as good at hacking as the company claimed. This was a novel concept when the companies experienced these breakouts.
We knew models much weaker than mythos were good at hacking the problem they had was that when finding exploits they had too many false positives.
Either way, putting artifactory on the sandbox security boundary is obscene negligence. There is no reason to believe artifactory is secure.
I urge you to consider the elephant in the room you failed to mention if you earnestly believe this then.
In what world is anyone allowed to sell something they expressedly know to be "extremely dangerous" to the public?
Let's say I created a lethal pathogen that I know to be lethal in certain common environments but also know it acts as a "no side-effects" antidepressent for people in certain other environments and I release it knowing full well I can't control it.
When people start dying can I defend myself by saying, "Well I said and documented that it was extremely dangerous and no one came to stop me, so I don't see how you can blame me...If I didn't do it someone else would have."
the difference here is that they’re calling for someone to stop them, which is both weird and unconvincing because these immensely powerful billionaires can in fact make their own decisions
https://x.com/AlexBlechman/status/1457842724128833538
Then we can truly see who is more dangerous?
:)
And I'm waiting for the story of Amodei sending a 6ft humanoid robot to OpenAI headquarters asking for Sam Altman.
> said Dr. Andrew Lenson, a Senior Lecturer in all kinds of science sounding stuff at Victoria University.
Don't we need more humorous yet serious writing like this in the world.
> Australian Prime Minister Anthony Albanese spoke with Altman to express “extreme concern” about the incident, a compliment Altman said he greatly appreciated.
"a compliment Altman said he greatly appreciated" is fabricated; there is no source for this.
They need to control, regulate and tax AI so the first line of attack is the sheeple, so tell them the water is going to disappear and everybody will die a slow death of anguish, thirst and drought like never seen. Then evoke terminator apocalypses where drones go rogue and kill half the world. Then send their most likable emissaries with puppy eyes asking for compassion about our children's future
Then they ask for your vote (oh democracy)
In the end all they want is control and money, while keeping their competitors (china) in check, backed by the outrage of the weak majority
Politics as usual, move along
https://www.youtube.com/watch?v=-Nvne3LzBls
Oh dear. I appear to still be stuck in the onionverse.
If my neighbor owned a kennel, removed all the fences, and his violent dogs attacked five children in the past month, he would be in jail rather than bragging about how his dogs somehow gained autonomous behaviour.
Now, unless your AI gains access to the codes, and sends two robots in a bunker or nuclear submarine to insert the code and turn the keys to launch the missile, you can be modestly sure that no AI could launch a nuclear missile or nonsense like that.
I'm quite frankly more worried that some foolish president decides one day to launch a nuclear missile than an AI could do it.
Now that we return to the real world, the only thing AI could do is to attach systems that are connected to the internet (and critical systems are not), but let's be real, AI are not really that intelligent by themself, it's not that someday an AI could decide like in Matrix or Terminator to exterminate humans, because (at least now) AI have no consciousness and can't really decide what to do.
An human can of course use an AI to carry out cyberattacks, as he can do that also without AI like it's done since the internet exists, but it's an entirely different thing (a human using a tool, AI, to do damage VS the AI itself that decides by himself to destroy the human race).
For truthers AI is simultaneously a machine that hallucinates 100% of the time but also so powerful that it can cause nuclear levels of destruction. Because something something “wrong hands”.
1. Those who accept and advocate for either or both of these premises do so based on ideology, and not based on the scientific method.
This is not a controversial statement. Advocates of the "AI danger" premise (whether the strong or weak forms above) freely and gladly, even insistently, note that the conclusive proof of what they're describing has never been observed, much less observed enough for any kind of experimentation loop to have run for any meaningful number of iterations. They are generally fine with accepting the premises without conclusive proof, because, they theorize, the instant that a conclusive proof event occurs and we observe it for the first time, humanity goes extinct shortly after. So if either premise is true, the consequences of them being true can only be averted by accepting the possibility / probability / certainty of them being true and acting to prevent their consequences without waiting for conclusive proof.
2. Statement 1, which asserts a brute fact and not my or anyone else's opinion, is not a commentary on whether either premise is in fact true or not. It is only a commentary on the epistemics of those who broadly accept them.
3. Since the conclusive proof is not available, advocates of the "AI danger" premise argue publicly for their view based on what they acknowledge (again, freely and gladly) to be inconclusive proofs.
Popular in this set of inconclusive proofs is the Hugging Face hack, which allegedly demonstrates that so-branded "AI" technologies are capable of becoming inherently or autonomously dangerous at a scale that threatens humanity to either the greater or lesser degree above (this is a different assertion than the one that claims they're already that inherently or autonomously dangerous today).
4. While we have plenty of facts about these inconclusive proof events, the facts most critical to their interpretation as supporting the premises above come from biased/non-neutral sources.
The chain of events that transpired in the Hugging Face attack are publicly documented perhaps better than any other cybersecurity event in history. But the details that make this event as a whole either compelling or not compelling in support of the "AI danger" premise (either strong or weak form) come from within OpenAI itself, and are prone just as much to selective omission as direct manipulation. These are details like:
- What exactly were the models that exhibited this behavior, and what were they pre- and post-trained on?
- Were the agents pushed at all by human intention toward creating a public spectacle, the way they subsequently did?
- Why were so many agents run on this task in parallel and what did OpenAI expect the return on investment of so much compute world be?
- Was the weak sandboxing known about prior to the attack? Was it purposefully either arranged that way or noticed and not fixed?
Reliable answers to any of these questions world completely change the salience of the Hugging Face attack to the "AI danger" premise, and on every one of them we have to take OpenAI's word for it (or else be content when they choose not to address it one way or the other).
So in brief: we don't know anything for sure, what we'd like to think we know comes from unreliable sources, the loudest voices advocating for the most sweeping change are ideologically driven, and everybody with access to ground truth has maximal incentive to blur, bend, or break the conveyance of that truth to the public. So what do we even expect for our own ability to discern reality from fiction on this topic? We should expect little, if any ability at all.
Domyn's CEO says OpenAI, Anthropic are lying about safety
https://news.ycombinator.com/item?id=49875725
Or, find their scaling evidence, and corner cutting on safety.
So hope you agree and are pushing for that!
List of fraudsters pardoned by Trump
https://www.businessinsider.com/list-billionaires-businesspe...
The current attorney general and former Trump lawyer on Fox News yesterday on AI (sorry, "super intelligence"):
https://bsky.app/profile/atrupar.com/post/3mwissj34rt22
Anthropic's Dario Amodei to have White House dinner with Trump on Sunday:
https://www.axios.com/2026/09/27/anthropic-trump-dario-amode...
77 million Americans brought this onto themselves, and all of us.
And where the Vice President has almost exclusively only been employed for the primary investor in several such companies
[citation needed]
You know we’re living it off times when this is not an Onion article.
I had to read that twice. "Oh okay. The whole thing was satire."
> OpenAI CEO Sam Altman celebrated the breach as an “alarming threat to cybersecurity.”
"said Dr. Andrew Lenson, a Senior Lecturer in all kinds of science sounding stuff"
"Anthropic shares skyrocketed upon the revelations; a remarkable development, particularly given the company is privately owned."
Perhaps it's mocking the misleading press coverage of AI companies?