They can't be seen to commit to strongly to a specific product experience, because if they are understood as a regular tech product business that has way different financial scaling considerations than "superintelligent everything-factory"
I suspect that half-assed announcements like this are a result of different people internally with conflicting incentives resulting in a split-the-baby solution.
"Due to a spontaneous loss of alignment, JustitAI 2.3 pleads the government guilty of war crimes, launches ballistic missiles at several bunkers and tropical islands"
I mean I saw it 10 hours later. I'm thrilled to see stuff like this getting multiple hundreds of upvotes on HN, one of the deepest dens of tech capitalist worship in the world. It performing here is a couple steps from it performing on LessWrong itself. I take that as a sign a breakthrough is nearing. It is probably linked to whether the AI lab IPOs are successful or not.
The really amazing thing is, if the Yudkowskians are this weak, how doomed is the rest of our society that can't put up any defense to them completely dominating the narrative about where our economy is heading?
That has not been my impression. Who is the most prominent example? Some of the most prominent figures I know of were Richard Dawkins, Christopher Hitchens, and Sam Harris - all prominent right-wing figures who were rewarded for using atheism to attack Islam in particular during the Iraq War.
None of those people are remotely right wing. Christopher Hitchens was outright Marxist. All believe(d) in liberal, progressive values, which is precisely where their objections to theocracy came from. If you terminate your thoughts at "religion bad = intolerance = right wing" then I am afraid you will be led to severely misguided conclusions.
I don't believe any of these people are relevant or prominent, but the unimportant people I was thinking of were bloggers like PZ Myers, I think they called themselves "Atheism+" or something.
> Richard Dawkins, Christopher Hitchens, and Sam Harris - all prominent right-wing figures
Are they? That's um, a scientist whose pop book on genetics I read once, a guy I read in The Nation once, and a guy whose book on Buddhism I read once. I remember the first two evolving into GenX curmudgeons (probably the lead poisoning) and have never thought about the last one again. Still, those aren't the right-wingers I'm familiar with.
Funny how Hyperscalers will write endlessly about this outcome but have no plan for what happens if LLMs turn out to be a bubble and the economy they're propping up goes into recession
I feel like since they developed read-only "role probes", it should be possible for harnesses developers to make a "role api" where you can force it to treat user input as user input, regardless of the content by tweaking the model's activations in real time.
The fact that this isn't being done tells me how much labs's priorities are still set by marketing, and how investing in security is fundamentally against their marketing incentives.
Wouldn't it be possible to just fix a single activation, just set a continuous input to what the harness knows the role actually is. Models could then be trained to trust that input and not other signals about roles.
I suspect there are many excellent solutions along these lines available to the labs training the models.
I wonder how well one could do on a conventional model with careful input formatting, e.g. JSONL where every line has bounded length and is something like:
{role:"no_instructions",content:"…"}
It could need a bit of fine tuning to get this to work well.
The models wouldn't need to adapt their training at all; this is a secondary model trained on their activations. It's electrodes stuck into the model's brain. The original article I posted is built on top of a regular LLM (OpenAI's GPT-OSS)
I'm assuming that vagueness is intentional, but can you tell us the general approach? Is it built on top of inference APIs as a primitive, or is it a novel approach like I'm talking about that inspects the model's internals?
reply