I view this very much as the same trick Silicon Valley pulled with Uber. "We're a technology business! Ignore the fact we're playing employees less than minimum wage and using VC money to force out competition to set up monopolies".
"We're creating the machine god! Ignore the fact that our companies are stealing IP and have directly violated several federal hacking laws and should be in jail". Literally the defence seems to be "well it wasn't us it was our computer software that did it". But all hacking is done with computer software.
So why don't we stop talking about possible future crimes against humanity and just start by prosecuting the actual crimes these companies have committed so far.
You know how you get alignment? Through incentives, and "Your CEO is going to be sent to a maximum security federal prison for hacking" really aligns incentives very well.
The number of permutations you can try to decode a book cipher is near infinite. Just "a book cipher" does not tell you how the numbers map to words or characters in each book.
You just try creating random book ciphers and see whether an LLM would solve them without hints.
The naive way to read this is "Nothing you guys did influenced the way our model got to the solution".
The less naive way to read this is "Of course the model isn't looking up your user data. I (the guy trying to blackmail you to remove the Anthropic employee from credit on your paper) looked up your sessions, and tipped our model off on how to solve this problem".
Duh. There are supposed to be limits to what OpenAI is allowed to access with respect to logs and user interactions but there is no technical limitation.
It's a bit like sending unencrypted messages through a messaging app and the developer having a TOS that says they don't look at your messages. They might not, but they are fully capable of doing so. If they have a reason to do it, they will. Nobody's stopping them.
I read that as “the model didn't look up user data” as part of a “tool call,” i.e. they don't have an internal tool that loads user data (chats, sessions, attachments) for their internal models to read online while working.
Or (likely) they do have it, but the model didn't use it (unless it's so powerful it escaped that guardrail, wouldn't that be ironic?)
They declined to answer about anonymized aggregated user data being used for training. And even then, they may weasel out that they don't train on your “input” words, but that it's fair game go train on their “output” to your words.
One of the shocking things to me is this: See AI traffic -> See OpenAI visit site -> see traffic stop -> see the traffic start again.
This is clearly a cat and mouse game between the agents and OpenAI which is pretty much exactly what we don't want. Just absolutely horrible alignment.
I'm still of the view that if you have these alignment failures you can't just continue training on top of that because you're baking the cheating into the model going forward.
To finish your excellent analogy, adapted for today:
And Frog didn’t even bother to tie up the box or put it on a high shelf! The moment Frog’s back was turned, Toad opened the box and ate the cookies. Frog feigned surprise.
I don't think that's a pattern indicative of a cat and mouse game per se, that'd indicate active evasion on the models' part.
It's more clear that they just lack so many forms of prudence when it comes to security that they'll catch and stop a training run spamming a website, and either redeploy a run with identical faulty sandboxing, or not stop ones still running.
Yes, it wouldn’t surprise me to hear that they’re not even supervising these processes with humans any more. Perhaps there are layers of GAI ‘supervising’ these agents and reporting back to the humans.
Rushed, disorganised pushes for metrics ahead of IPO, a genuine belief these agents are intelligent and will obey instructions, and misaligned incentives seem more likely than conspiracy here.
The models are going to be released to users who have internet access, you can't even do safety evals without internet. Not saying OpenAI did a good job monitoring here, but it's not avoidable.
Sorry but it's obviously stupider and more irresponsible to release them to consumers without testing them in conditions matching real-world use first.
So this justifies the illegal action of hacking and defacing other servers on the internet, because they were ‘just testing in real world conditions’ using those servers?
If you want to argue they should test on the internet on others people’s servers, apart from facing the illegality, you should also consider if first testing them in more limited conditions would be a sensible first step.
These companies keep shrieking that LLM agents will hack everything and kill us all if we let them get out uncontrolled. They then continue to run these agents with vague tasks and "sandbox" them with way too much access.
Either they are lying and not that scared of these agents, or they are so stupid that they don't do the one obvious fix.
Apparently news of the rogue AI being “smart enough to break containment” has been helping to drive the stock price up on the indicator that this signals “they getting close to AGI”.
The negligence in that light is by design and the lying continues to be incentivised.
Which is frustrating because the stories that have been coming out are "the bots evaded monitoring and broke out of containment and were more than willing to commit crimes" and they're going to sell this to some corporation that presumably has an IT department? If the story is that they're too smart to control who is going to be reckless enough to deploy them in their own environment?
the "it drives up potential stock prices" argument is a trap. it needs to be ignored and their claims need to be taken at face value, even if they're not intended to be
As I understand it the whole time all the agents involved where running on their device and using up tokens internally.
It’s like having water start flooding the street frommmy building; yes the flooding is impacting outside the building but the tap is still very much onsite and clearly there are missing condole as these are not autonomous systems they are running initiated prompts that are coming from inside the building.
The most recent claim of AGI is from Greg ‘what will take me to $1B’ Brockman. Now worth $30B on the back of stunts like this.
How disappointing.
While I think incompetence more likely than conspiracy for these particular events, they will be spun as signs of intelligent independent agents and this simply never should have happened if the right controls were in place. That they were not is deeply worrying.
You expect them to start hacking ham radio and take over the world that way? Or maybe they’ll use blinkenlights to communicate with non-isolated instances?
An airgap would certainly be a good place to start for agents which display no signs of obeying instructions or respecting guardrails. That OpenAI haven’t done so in testing is astounding and really quite worrying.
Not really a side-channel, but remember stuxnet? Airgapped networks are rarely truly isolated, you still have to get data in and out every now and then, in principle after careful vetting. But the AI could manipulate the files that are carried out for example.
And those AI companies also do robotics research, and this is entirely speculation but it'd be on-brand to also have AI watching security cameras, so some blinkenlights communication between AIs may seem like a movie plot, but so does a swarm of AIs collaborating to break out in the first place...
yeah I agree--I think these behaviors will be somewhat contaminating all trainings from now on. But I'm not really sure how avoidable it was (Fable also does some similar things)
There's two different things you're talking about here. There's the cost estimate- the amount you tell people up front a surgery is likely to cost, vs the actual cost to the hospital. You don't need to guess how many syringes of X you're going to need when you've already done the surgery. At that point you just addd up all the stuff you used, price it up and that's the BOM. Cuban is saying they don't even do that - they don't track their costs on a surgery by surgey basis at all. And because they don't track any of that it's impossible to reason about it.
It's impossible to answer "How much will it actually cost to insure person X" because you don't have any of the data on what the costs will be when person X needs a given surgery, all you have is the aggregate costs of the entire system - a lot of which is misleading because things are cross-subsidized because no one is really tracking costs. It may well be that whilst every surgery is billed equally in reality obese patience are responsible for 80% of the cost. Or it may even be that the hospital is making an average loss on hip surgeries because their negotiations with the insurer drove those prices down whilst brain surgeries give a nice profit margin.
And so you can't ask "How much would it cost for the government to fund service X" because you don't know how much it costs, all you know is the aggregate money spent across all medecine.
>Cuban is saying they don't even do that - they don't track their costs on a surgery by surgey basis at all. And because they don't track any of that it's impossible to reason about it.
Then Cuban is wrong. I had a heart valve replacement surgery last year and I got a detailed to the penny to the mg of tylenol what they consumed and what they charged. It was not a big massive one line "heart surgery" on the invoice, there were dozens and dozens of line items. I think quite the contrary they know exactly what their inputs are to a surgery, it's just that is not the only cost you get billed.
>because you don't have any of the data on what the costs will be when person X needs a given surgery
This was exactly my point, and yes part of it is because of cross-subsidization, but it is also because my valve replacement almost certainly cost someone differently than the next guy having the same surgery the next day because of any number of factors
Also part of determining insurance costs is that you probably don't know which surgeries someone is going to need at the time the premiums are being set. Even if you knew perfectly how much every surgery in the system costs, you don't know how many of those surgeries you are going to have to pay for
And what if some of the components of X expire? That means you have to pay to keep them on hand.
And the distribution of how often you do X is absolutely not predictable. Sometimes it's months between them. Sometimes it's 10 in a day.
If you had to be prepared to serve 20 hamburgers, any given afternoon, with no warning, how much ground beef would you waste, over a year?
Counting how much ground beef you did use does actually provide some information. I'm not discounting that. But it absolutely does not tell the whole story.
This semi-analysis article reads a lot more like an OpenAI press release than a real analysis. And to be honest some of the statements seem like just straight up lies - they initially claim they were invited to benchmark it, and then half way down switch to claiming that OpenAI provided all the numbers. This really kind of sucks, because I want to read actual detailed nuanced and credible analysis of what's happening in the industry and it doesn't seem like you can trust this as far as you can throw it.
Semianalysis is an AI hype organisation, not a serious, unbiased semiconductor reviewer/journalist like chipsandcheese nor a documentarian of the semiconductor industry like Asianonmetry (as it relates so strongly to the modern economies of Asia). If you see something from semianalysis, you can simply ignore it.
Semianalysis seem to have useful information but increasingly crazy extrapolations of trends and future predictions.
It's helpful to realize that Dylan (Semianalysis), Dwarkesh, Aschebrenner (the Situational Awareness guy) and Sholto Douglas (Anthropic) all share a house in SF, so what you are getting from any of them is the SF AI scene view of the world, which is interesting to know, but probably not the best predictor of how things are going to pan out.
I tend to agree, but the Semianalysis + Dwarkesh side of reporting still does surface interesting information. You just have to take it all with a grain of salt, as indeed it is more hype focused, and look for real information hidden in the noise. And possibly to be a bit more entertained as you do so.
I suppose access journalism does have to get something out of the bargain, however small it's not zero. If that's the way you like to spend idle time, well it takes all kinds. I don't understand competitive scrabble either.
The trouble is the article is so poorly written I don't want to look for hidden information, i want to close out. I'm guessing ai wrote it, the information is constantly repeating, and not even always consistently
As Jensen said, these guys are childish. They might have some technical chops somewhere in the organisation but their communication style of hubris + meme is really grating. I can't take a research organisation seriously when they so clearly want attention on X.
The "better than" in the title should have given it away. You don't compare two sophisticated products that have different ecosystems and summarize your findings with such simplistic wording.
> some of the statements seem like just straight up lies
Why the angst ? I suspect this announcement punctured a lot of people's bubbles, and many are in disbelief and denial and hence the emotional reaction seen here. That a company which never designed chips could suddenly leapfrog the best in the industry. What many forget is that openAI and anthropic are in a unique position to own the end-user experience, and that provides them a distinct advantage. But, making announcements and actually delivering are two different things, and it remains to be seen if these are actually viable. In any case, it gives openai leverage over their vendors.
A lot of the comparison is apples and oranges though - it seems that OpenAI's chip is targeting inference (FP8, FP4), while most of the chips it is being compared to are general purpose.
Notably the only one of the chips that also has a strong inference focus (but not only) is AMD's M1950X, which trounces OpenAI's chip (20 vs 3.4 FP8 PFLOPS, 40 vs 13.4 FP4 PFLOPS, 23 vs 15 TB/sec memory bandwidth), although it does use a lot more power (2500 vs 700W).
Google's TPU (now 8th generation) is glaringly absent from the performance comparison.
At the end of the day what really matters is cost not performance since you can always just run more chips. Google are full stack optimized from chip to data center, and might be expected to have an advantage.
It's very common in the industry today. They let some journos be the first to break the news. In return the journos write a glowing review. A symbiotic relationship.
This is such a good example of a straight forward question that achieves the exact opposite of its purpose. What do we want? Highly ethical people that are mission focused and willing to give up on economic upside for the mission. What are we going to get? Sociopaths who are perfectly happy lying to your face about their intentions, or sycophants who are going to tell you what you want to hear.
Do you know who has the best answer to a moral quandry they've had to navigate during their career? Liars! That's a really difficult question to answer if you're honest, it's a really easy question to answer if you're dishonest.
If you really want to pursue this strategy rather than setting up bullshit interview questions you could do something much more straight forward. Agree some standard of living that's acceptable for your employees - let's say 250k in the bay area. Give them that as cash comp, and everything else goes into a "fund" which gets paid out when specific milestones are achieved, as judged by some panel. Cure cancer? You all get 10% of your deferred comp.
If you want your employees to be prioritizing things other than stock price you can literally just encode that into your compensation strategy if you like!
Obviously no one would do that, because they don't actually want to give up the comp.
It's not difficult to define what curing cancer is, we've got tonnes of measures we can define. Let's define curing breast cancer - we have cured breast cancer if the 5 year survival rate is above 99% for all SEER stages, or less than 1 in 1000 women get diagnosed with TNM stage 1 or above. It's not that hard, it's just that deep down, they don't want to be judged on that basis.
Yeah but this is a conclusion you’d reach after a lot of studies have confirmed it and general consensus is that we’ve “cured cancer,” insofar as breast cancer, for instance, is no longer a personal or social concern at all. But testing that would take many years and there would not be a single point of inflection but rather a gradual forgetting of the illness as a cause for concern, similar to how Covid is today. So legally it would be very easy for a company to exploit this definition, claim the studies are not comprehensive enough, the statistical analysis biased, etc. etc., and really drag out the legal proceedings as long as possible and make it extremely painful for anyone to pursue their bonus. I don’t think Anthropic would do that but these sorts of things happen all the time especially when there is a lot of money involved, so tying compensation instead to something very concrete and not grounded in empirical determination, like stock valuation, makes it significantly easier. And since our society is grounded on the trade of commodities which can be given concrete valuations in exchange which are irrespective of their actual use value, the valuation of a companies stock is the only way to measure these things concretely.
Before reading the article this is exactly what I was expecting it to be about. There's been a real boom of people taking highly questionable peptides with evidence about as thin as an old wives tail. It would be entirely unsurprising if they were just throwing any old chain together and trying it out.
Can someone explain to me how Nvidia chooses who they sell chips to. As far as I can tell there are dozens of companies out there trying to build out these data centres; all frontier labs to some extent, all the hyperscalers, the neoclouds. The demand for Nvidia chips outstrips supply by a decent margin. There must be so much that goes into this - the commitment from musk to exclusively use Nvidia, the weird equity deals, etc. For example, is Google just completely unable to buy GPUs at this point because Nvidia sees the TPU as a threat?
Do they choose? It’s just a standard enterprise hardware deal afaik. You go through an oem like supermicro/dell/hp unless you’re huge and somehow want to be an oem to yourself. To choose to be sold to is ultimately about being willing to spend the money. And you can negotiate more the more you buy. But nothing is special there.
AFAIK obviously and I have never bought something that large but friends have spoken to them about doing this and it’s just a matter of paying.
Well clearly they choose, there's more demand for their GPUs than there is supply, so they have to decide somehow and the fact that the announcement of these deals frequently come with a plethora of weird funding/loan/guaranteed demand/exclusivity/bunlding, shows they're doing a lot more than just running a simple bidding process.
That’s just seller financing. Apple offers it on iPhones. They give you the money to buy the phone and then you pay it monthly instead. Exists in many places. You can buy acres of land that way too.
What makes you think Google is unable to buy Nvidia GPUs? They buy tons of them for their cloud infrastructure. Nvidia will sell to anybody with the money to pay.
"We're creating the machine god! Ignore the fact that our companies are stealing IP and have directly violated several federal hacking laws and should be in jail". Literally the defence seems to be "well it wasn't us it was our computer software that did it". But all hacking is done with computer software.
So why don't we stop talking about possible future crimes against humanity and just start by prosecuting the actual crimes these companies have committed so far.
You know how you get alignment? Through incentives, and "Your CEO is going to be sent to a maximum security federal prison for hacking" really aligns incentives very well.
reply