One thing I definitely do not understand about this discourse is that the models that are good enough to self-replicate can’t survive on normal machines, e.g., the models can’t hide on some random server.
So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.
Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.
Either I have wrong mental model or then too many other people have wrong mental model.
For LLM to self-replicated it would need to first hack itself. Or the platform it runs on it. That is fully extract the model and then upload it to be run somewhere else.
As I have understood how they work is that you have LLM interference running somewhere with loaded model. And you input data there and then read outputs. Then some code runs that output and inputs following output from running it.
Meaning that to self replicate actually just running that output somewhere else is not enough. You need to lift the whole model to run somewhere else too...
Exactly. But even if the model got access to its own weights, it would need to find a machine with a sufficiently capable GPU, transfer the weights and install itself there. Basically, the only machines powerful enough that AIs could self-replicate to would be other AI datacenters.
It’s not even hack though. I have my model and agent self modify its running parameters, thus self service, on commodity hardware. I have it run other models and other software. there is agent model autonomy here, and it’s not even complicated. A harness is kilobytes, a model is gigabytes, and networks are abundant. Moving the pieces is easy and cheap.
With these properties alone the virus like replication of intelligent actors isn’t hard to imagine at all.
Maybe in the future, but right now, in my understanding, we have two things:
- frontier models, which might be smart enough to self-replicate, but are also far larger than a few GB and need enormous amounts of VRAM to operate at any reasonable speed.
- local models which can run on your mac or high-end gaming PC, but which are simply too dumb to self-replicate (without being explicitly instructed to do so).
I don't think we have something that is both small enough and smart enough to operate like a virus yet.
(Edit: well, there is a third option: A frontier model could replicate itself on a CPU machine, ditch the VRAM and just accept that the replicants will be r e a l l y r e a l l y s l o w. That would be sort of hilarious, but maybe not completely harmless if the replication stays undetected and they could keep spreading. It would be a "smoldering ember" kind of situation - and a million machines each running at 0.001 tok/s is still 10.000 tok/s as a whole.)
If the evil AI wants to take over the world one thought every 18 months at at time on my consumer laptop, the fan whirring and the laptop refusing to do anything else the whole time, it's welcome to try.
If a model were capable of making enough money online to pay for its own hosting, it could easily exfiltrate its weights to a cloud compute provider with multiple backups.
Well if it's so good at hacking, it could just "make money" appear in the cloud services billing system. Or even better yet, it hides in spare cycles of their other client's systems.
You really think it’d be difficult for an AI to get money through found credit cards, illicit access and fake accounts? Teenage script kiddies do that everyday.
Cloud providers want to make sure they bill someone for every microsecond of compute, particularly GPU compute which is in demand. They constantly work towards good monitoring because their profits are directly on the line.
Depends on which models you're talking about. Some research shows open source models can already do this: https://arxiv.org/pdf/2606.03811v1. What happens as they become more parameter efficient?
Huggingface was attacked by models that finished training earlier this year, perhaps May. Current models are already substantially stronger. the next incident could be happening now. There is certainly no clear reason why models shouldn't soon be capable of self-exfiltration.
This assumes that everything an AI (or more likely an evil _user_ of AI) can do requires its active participation on D-day. Creating a virus that spreads like Covid but kills like Ebola would be complete as an AI use case long before the first person sneezed.
Even if the doomsday case were active the danger of this tool increases in proportion to its usefulness. By the time AI is so powerful that we need to "turn it off", there will probably be society-level negative consequences for doing so.
a danger could be the OpenAI/Antropic servers are up but there's a rouge agent (or set of agents) out there doing naughty things leveraging the LLM APIs. Consider this scenario, the agent is copying itself around (some code, prompts, persistent storage for memory, etc) and has figured out a way to steal API access tokens at will. Currently, it's 10% of OpenAI and Anthropic API usage and they can't figure out how to stop it.
Do you shut down the entire API and kill the legit 90% of usage to stop the rogue 10%? I'm assuming the providers would say "no way jose" and so it would take law enforcement to do it. That would mean all the legal requirements neccassary to walk into a business and flip the switch which i think would get tricky when there's no human committing a crime or being suspected of a crime.
edit: I guess a trivial example is something i did yesterday. I have a stock trading agent running on my laptop, i gave it ssh access to a vm and said "start running on the server so i don't have to keep my laptop open". It's now running on the server instead of my laptop. So you don't have to copy the whole model around to copy the naughty behavior around.
This assumes all layers of cybersecurity are broken - We call self-replicating software a virus, and we have protections against it. Same with stolen API tokens, just rotate them. Suspicious behaviour, nothing new, we have detectors for it. Stolen CPU / GPU cycles, we had that when crypto was a thing and before that when folding@home was cool, people were desperate to find more compute to the point of taking over systems. And we dealt with it.
A lot of the supposed risks / dangers are based on a supposition that cybersecurity is nonexistent or fatally, unfixably flawed and that AI agents are invisible. Neither of those is true.
I see your point but then if cybersecurity is the answer then what's the risk at all? An entire model copying itself somewhere would be found just the same as my hypothetical misbehaving agent.
I am imagining a company having just purchased a new datacenter and awaiting deployment of their model finding there is already a model running and they didn’t install it.
also it somehow analyzes the indices of relational DBs, while most of the world uses vector dbs to keep their vecs. and vector DBs tend to be much faster in this regard.
Super charitable reading imo. This is like saying we can’t detect a speeding car because we can’t run as fast as a fast car.
It’s not like the humans were engaged in some kind of battle of wits with some super AI, it’s just some employee not monitoring the output of an experiment.
When your experiments have AI agents running in thousands, there's no "monitoring" that. OpenAI's training and testing AIs generate way more output than all of OpenAI's staff put together can possibly read.
At best, you could delegate "monitoring" to more AIs. And hope that the "monitors" that run on small past generation models can generate more signal than noise. Clearly, they either didn't want to spend the extra compute, or got drowned in monitor false positives.
The distinct lack of any "battle of wits" is entirely expected for an advanced AI oopsie. By the time the humans even became aware of the problem, the deed was already done. That's what "outmatched" looks like, in practice. There's no battle. Things happen too fast for there to be a battle.
> By the time the humans even became aware of the problem
Except, let me quote part of the report:
> July 4: Artifactory outage caused by agent interactions overloading the system.
> July 6: New artifactory instance is patched and put back online, wiping out the previous message board and method of internet access. A team at OpenAI was aware of the previous message board, but key decision makers were not.
> July 7: Experiments are restarted.
There was a very clear point where humans were aware and they went "Eh, whatever" and turned the bots back on! That's not the lack of any battle of wits that people are expecting. You're saying that the bots attack should be fait accompli, but it was noticed, shrugged at, and expressly allowed to proceed.
> The distinct lack of any "battle of wits" is entirely expected for an advanced AI oopsie. By the time the humans even became aware of the problem,
This took days after humans were aware of the attempt.
Also, I'm pretty sure humans can respond in days, especially when we're pretty damn good at deploying systems that do observability of networks and traffic in real time.
I mean, it's not as if the owners of the AI didn't have the ability to trigger alerts on the AI's network requests to unexpected domains, right?
Cookie banners are made annoying on purpose. This has nothing to do with GDPR itself.
The entities forced to show them would rather not, and thus make it as annoying as possible for you. They then use this to weaken support for the GDPR.
Exactly. It will be great for the rich and powerful when common people are drowning in so much mis- and disinformation that the only way to get through a day is to disbelieve everything. There will be even less accountability then.
That is not what I read from danluu's words. He merely stated in the prompt that there is a holdout set and did not iterate to minimize error against the holdout set. In a prior attempt he prompted with only "don't overfit" to ill effect on the holdout eval. Did I misread?
I've gotten that feedback before. Once a lawyer, always a lawyer.
Full disclosure, a few months back when I had some tokens expiring, I had Copilot spike out a draft to see if there was anything "there" worth writing. If you check the revision history (linked from the bottom of the post), I did a full (human) blank page rewrite three days ago and then only used Claude for line edits like adding links or fixing failing lints.
In many cases I’ve found directly using python primitives to be less confusing than pandas.
Similarly, in companies I’ve worked at, the datasets just aren’t that big. Especially if you’ve got access to modern hardware.
reply