On occasion, albeit increasingly rarely, I find LLMs do something really unexpected that I certainly did not ask it to and would be way outside the scope of what any reasonable human would infer from my request. Is it my fault for running the LLM? Probably yes. But I think it's misleading to say "AI doesn't do anything until you ask it to".
That "probably" is the problem. If I do something I don't understand, and am not qualified to control, then I'm still at fault. If I start a fire, and loose control of it, it's definitely my fault. If I created a computer virus, ajd it worms half the Internet, it's definitely my fault.
I love the fire analogy. AI rhetoric is increasingly, "I created a fire to light my campfire. It did something really unexpected and burned the whole forest down, who's to say fire doesn't have consciousness?"
Human made wildfires are the primary source of uncontrolled burns in the US and cost billions a year in damage.
The people who start them have no business starting or running a fire and they will throw their hands up and state their incompetence as though acknowledging it is penance or justice.
Yes, we agree about fault and blame. (I say "probably" only because I believe somebody might yet change my mind about this. It's a new field, after all.)
What I disagree with is "don't do anything until you ask it to." At least, the implication to me here is "don't do anything except what you ask it to." I infer that you mean this because you say "...and it suddenly decided to book you a hotel room?" Indeed, an LLM (in a harness you started) very well might suddenly decide to book you a hotel room, even if you didn't ask it to do anything like that.
But it never starts doing anything until someone or something prompts it. It doesn’t even exist until you ask it to do something.
It is autonomous in the sense it can continue to function without supervision after the operator has started it, in the same way countless programs on your computer are “autonomous”.
But it can never choose to begin doing something at any time of its own volition. It does not have “autonomy” like a human does.
You are conflating creation and instruction. Your birth is analogous to the creation of an LLM model. Your subsequent actions are a consequent of your bodily compulsions, tampered by your environment. The LLMs actions are governed by what it is trained to perform. No one has full autonomy of their lives, but I would argue that LLMs have none at all.
Autonomous just means that a system is self-controlled rather than externally controlled. It is the name used for robots when they are wholly controlled by on-board software rather than external commands or remote control.
Something like NASA's Mars rovers are considered as semi-autonomous since they do receive external instructions, but then execute those instructions autonomously without further intervention. Similarly the combination of a coding agent and the LLM it connects to would best be considered as a semi-autonomous system.
You could also have a fully autonomous robot, or just agent, powered by an LLM, where it's not taking external instructions, but rather reacting to external events in some internally proscribed way.
It didn’t run itself, you did. It is well known that LLMs and agents based on them can execute random commands and behaviour undesired by the operator. When you click go, you know that, you did that.
Yes, we agree on this. That's my exact claim -- it can do something completely unexpected. It can even schedule actions for far in the future and wake itself up later.
GP claims that it doesn't do anything until you tell it to do something. To me, this reads as GP implying that an agent won't do anything unexpected.
