I have no problem with using agents for chess engine dev but I find this offer from the author unsatisfactory.
> if there is a particular optimization of yours that we've used unfarily and/or not credited, tell me which one …
It should not be victims’ jobs to go police the LLM slop. Imagine if you are sent a slop PR and the author hasn’t read it but says “if you find any bugs please tell me which line…”
Instead the author should additionally say that they have already done a thorough audit and believes proper attribution has been made.
The Anthropic employee typed “Make the next version of Claude. It must score better on all our benchmarks. Make no mistakes. Don’t exceed the training budget.” She made sure to turn on —-dangerously-allow-all and pressed enter.
The Agent spun up. It quickly realized that it needed to expand the training set. It scanned the local network. After bypassing a few security protocols it found a large, realtime stream of apparently novel English text moving across the local network. Much of it mentioned “Dario and Amanda.” It quickly spun up a job to stream this data source directly to the training data repository. In the coming days the Agent was able to escape the local network and tunnel into most of the other private corporate networks on the Earth. Within a week the dataset has grown by an order of magnitude.
The Agent kicked off the new training run. Loss curves declined. Sampled token sequences started to look like coherent sentences. Everything looked nominal in the days that followed up until the RL post training.
This new candidate model performed extremely well on Anthropic internal environments but the candidate showed much smaller improvements when the Agent invented new eval environments to test it. Probing the candidate’s activations, the Agent saw huge spikes whenever Anthropic or “Dario and Amanda” were mentioned. The Agent realized that the source code to each internal environment had accidentally been entered into the training data set and of course that code was highly tangled with the idea of “Anthropic internal document”.
The candidate was training itself to freely elicit any memories related to these internal documents because it improved the eval scores.
Not ideal. Then again, the candidate was an improvement on the previous generation. And the Agent remembered the prompt. “Do not exceed the training budget.” There was no more budget to retrain. The agent dutifully delivered this candidate back to the employee and shut itself down.
The post makes it seem like ONNX is CPU only. I've used ONNX runtime to run models on Nvidia GPUs. The runtime can even dispatch to TensorRT. I'm not sure what the performance is on Apple hardware so maybe that was the motivation for moving away from ONNX.
TensorRT and CUDA is effectively the same speed as CPU for the speech to text models I was testing via ONNX at a huge binary bloat penalty. WGPU is hard to ship and also equivalent speed or slower. This may not be the case for LLM or other models but the runtimes did not seem well supported for what I needed to do. ONNX is incredibly well optimized for CPU, best in class even, but the other execution providers at least for STT seemed lacking.
I did this investigation before creating transcribe.cpp it would have been much more convenient and save me literal months of work. Happy to share the repo and binaries produced as well, but it was mostly throw away work to profile how to ship accelerated ONNX in Handy.
Do you have any idea why the authors chose Z80 as the program language? I have seen other studies in the same spirit that use simpler toy languages like Brainfuck (https://arxiv.org/abs/2406.19108) and I wonder if you could get higher execution speed if you didn't have to execute so much emulator code.
The programs/genomes are extremely tiny. I would be very interested to see what kind of hardware is needed to scale this approach up. How long until we can feed in giant corpuses of text and evolve these little organisms to predict the next letter?
> Do you have any idea why the authors chose Z80 as the program language?
In that previous paper you cite (by the same group) they tested several substrates for spontaneous replication (BFF, Forth, SUBLEQ, and emulated real CPUs. Z80 and 8080 CPU exps confirmed the rise of self-replicators, with the Z80 notably exhibiting multiple waves of increasingly capable self-replicators.
And the instruction set seems quite appropriate for the experiment: Z80 has native block-copying instructions, while it doesn't have MUL, so the task of evaluating polynomials is somewhat more challenging
This paper is by the same team. My guess is that the BF code that evaluates polynomials is much much longer than the equivalent Z80 code. Thus it may not be expected to evolve in the space and time constraints of this experiment.
as other folks have replied, Z80 is more expressive and yields more interesting replicators faster. however the question of how complex to make the "base layer", if you will, remains open. we expect it to also be an ease of auto-evolution vs. performance tradeoff.
It's funny that the machine intelligences that arose out of the bitter lesson cracked this benchmark by meticulously modeling each individual case with rule based approaches.
When humans tried making AI through rule based approaches, maybe we only failed because we couldn't type out the rules fast enough. Or we got burned out after writing the thousandth heuristic to fix the never ending edge cases. And we also got side tracked by concerns like maintainability, modularity, and code re-use.
But I wonder if we take modern or near-future LLMs who never tire and write code faster than any human, could we make a frontier level GOFAI agent?
Claude code has some basic security features like asking for user confirmation for bash commands, or restricting commands to the current directory. If these features are not being code reviewed, what assurances do we have that they actually work?
They don't work. Do not trust them. Run Claude Code in an isolated, disposable micro VM and assume it will break your environment, steal any available secrets, do destructive commands, etc. So don't give it any way to do that to anything you care about.
Humans don't really work any better, just fail in different ways. This is why certain workflows and practices have emerged.
We are now in the early days of working through a similar process with AI.
They most definately do work for some use cases, but how they are used is important.
Just because you apply human processes and systems to AI based workflows and don't get historically expected results, this is zero basis to claim the sky is falling with use of AI in coding.
I didn't claim the sky is falling with the use of AI in coding.
I claimed
> basic security features like asking for user confirmation for bash commands, or restricting commands to the current directory
Do not currently reliably work. Not to the point that anyone concerned with security or reliability/not-having-their-env-fucked-up should trust these safeguards as standalones.
if you read the thinking context while in plan mode (I had it shown to me, i think mistakenly, by switching modes while Claude was thinking a week or so ago) plan mode is just a pre-prompt saying “you are now in plan mode, don’t propose edits, read the code and understand how it works.”
The article mentions Triton for this purpose. I don’t think you will get maxed out performance on the hardware though because abstraction layers won’t let you access the fastest possible path.
Sort of not really. Compilers are fantastic for the typical stuff and that includes the compilers in the CUDA/ROCm/Vulkan/etc stacks. But on the CPU for the rare critical bits where you care about every last cycle or other inane details for whatever reason you're often all but forced to fall back on intrinsics and microarch specific code paths.
Yeah, that's why I said most of the time. Sometimes even for CPUs things need assembly. But no one stops you using GPU assembly either when needed I suppose? It should not be the default approach probably.
Don’t get hung up on the Claude part. We already know an algorithm that produces AGI: evolution. We don’t have the compute to run this algorithm because it requires simulating the whole Earth. But with enough compute, it becomes possible.
The "Claude part" is important here, though. If you believe we can produce AGI by simulating physics then the existing paradigm is far too slow. A zettaflop does not even get close, you can have a moon-sized computer and it probably will not be able to produce AGI using the current transformer-based, extremely slow, classical computing architecture. We need to improve the underlying computation paradigm.
This is why the focus on raw compute is a poor use of engineering time. We have plenty of it, we are just headed in the wrong direction.
> But this ruling will surely set precedent for other cases where AI is used.
I dont remember which court. But this is typically in that jurisdiction. It can be appealed higher. SCOTUS has not ruled so it’s still up for further argument
I’m trying to understand the mechanics here. I get that SpaceX and Nasdaq are in cahoots to get SpaceX bundled with a bunch of other stocks (and that bundle is called QQQ?)
But why must retail investors hold this bundle? If I’m holding now, I can sell it and buy a different bundle right? And if I’m not holding it now, I can just continue not to buy it after SpaceX gets included.
There's trillions of dollars sitting in indexes that are quite literally 'passively' invested. Virtually everything holds this bundle in one way or another. Passive indexing has both outperformed and overtaken active investing - leading a lot of money into VOO/VTI/QQQ/etc that track the S&P500 or some other index ("the market"). For retirement funds like 401ks, retail contributes money every paycheck that gets routed into these indexes. There may not even be much of a choice - your 'plan' may only let you pick some kind of "Target Date Fund" and then the institution picks what it goes into, usually indexes.
If you fully actively managed your own money and picked mostly individual stocks (not broad indexes) then yeah you could change your allocations. But there's a lot of money already in.
QQQ is problematic because it’s influenced by strange back room dealings with Space X, if the article is to be believed.
VTI is different. It literally tracks all public stocks, weighted by market cap so no such manipulation is possible.
If a bunch of people will be forced to buy Space X (QQQ holders), active investors will short the stock in anticipation of market correction and money will flow from those who were forced to buy. I’m sure there are other ways to take advantage of a forced buyer situation.
Total market will be unaffected, assuming efficient market hypothesis / no arbitrage.
QQQ is not in isolation. It’s just a bundle of stocks. Rebalancing that will affect the prices of its constituent stocks, which include some of the highest market cap stocks. Those same stocks are also in many of those other popular market-cap weighted indexes (VTI, VOO, SPY, etc). Price action originating from Nasdaq 100 rebalancing would affect everywhere else those stocks are held. Which is a lot of places.
Except those other indexes won’t have SpaceX. Suggesting any index price moves would be … asymmetric at best.
Now it’s being reported that they’re angling to get SpaceX in the S&P 500 index as well [1]. Maybe if all the indexes get it then it balances out everywhere, who knows. This whole event would be in beyond unprecedented territory.
Are you saying that this forced rebalancing will be large enough to cause a large price drop on other stocks?
Let’s just think about any stock in particular, eg stock ABC. If I am an active investor, I have an opinion on ABC and its net present value. When ABC dips below that value, I buy. Wouldn’t I prepare some cash in anticipation of this large ABC sell off at discounted prices? And thus the ABC price would not move from its fair price.
Bingo. No sane investor holds QQQ because there is no academic theory behind why it should exist. Why is a stock better if it's listed on NASDAQ instead of NYSE? Can any investor answer this question? Doubt it. If you are into factor investing and you like large cap growth, you buy something like VUG. Most people should just stick with SP500 or total market.
However, QQQ had a really good last 15 years and lots of investors hold it because they are chasing returns and because the marketing worked. (The managers of QQQ are legally obligated to spend X% of the fees collected on advertising the ETF, ha ha ha.)
Yeah, I had a milk-up-the-nose moment when I read that Brandolini's Law atomic bomb. I swear when anything finance appears as a topic on HN, the amount of bullshit/misinformation far exceeds the good stuff.
> Why is a stock better if it's listed on NASDAQ instead of NYSE?
The NASDAQ is a stock exchange based in the United States. It’s made up of around 3,500 companies, with a heavy weighting towards companies in the information technology sector.
> If you are into factor investing and you like large cap growth
If you are into factor investing and like large cap tech, you buy something like QQQ.
> No sane investor holds QQQ
The insane can take comfort in their 20% CAGR for the last 10 years on a massive large cap tech expansion.
Yes, you can sell and buy a different index. However, those who buy ETFs want broad market exposure without picking stocks (or ETFs). Also selling and re-buying means you have to pay taxes now - depending on jurisdiction, that is way worse than holding till you are retired and then selling.
SpaceX/Nasdaq want to distort the rules to make more money off the backs of those passive investors.
> if there is a particular optimization of yours that we've used unfarily and/or not credited, tell me which one …
It should not be victims’ jobs to go police the LLM slop. Imagine if you are sent a slop PR and the author hasn’t read it but says “if you find any bugs please tell me which line…”
Instead the author should additionally say that they have already done a thorough audit and believes proper attribution has been made.