Terra has not been able to do any of the technical tasks I've asked of it correctly. I'm surprised others get use out of it. Anything below Sol high tends to give me mostly unreliable results. I'm using codex as my main harness but maybe it performs better with a different one.
With Pi as a harness I've been using OpenAI models as a worker with an Opus 5 (in Claude Code) planner. I've only had a few issues with Terra High/Medium and absolutely none with Sol Medium+ on a fairly complex Rust project that targets Linux, Mac, Windows and Web, with plenty of nasty FFI, VMs, remotely debugging systems, among some other things within a monorepo.
I think the key is to give them a nice assortment of self-verification tools, an AGENTS.md or reference document that they're encouraged to routinely check, and asking the planner to be thorough with the ACs but give the model some space.
The planner routinely finds issues with the worker's output, but that's what it is for.
the breadth and width of the universe of oneshot challenges are all arbitrary. It's unsurprising different workflows oneshot better than others.
All the more reason to favor local models under your control, as once you find that sweet spot model, no one can change it, upgrade it, align it, take it down or otherwise harm the time investment you made it making it your own.
I can't really believe no one understands, after decades, how valueable a rock solid development environment is.
Exactly! I am not opposed to cloud-based models, but I do only stick to open-weight models because I know I can move my whole stack to local (given enough hardware) and continue development without any of the LLM interaction contracts being broken.
I would like to see some development where proof of authenticity certs are generated alongside the actual output of the model. Prove to me (or at least claim to me liable to breach of contract) that this output was generated by FP8 DeepSeek V4 Pro 0813. Not some cheaper quantization of the model.
Right now, sol-xhigh is my favorite model. I feel that Opus 5 is dumber than 4.8. Fable is too expensive to do anything (limit of $50, started a prompt at $25, ended up at $75, is bullshit, but at least it's "free credits").
DeepSeek is okay for random API-based stuff, as it's cheap.
Local open models running on a 5090 are hit or miss. I feel that most GGUFs/quants are awful...
I am still on Opus 4.8, with a custom built harness and it works very well even on multi-repos, across stack, deep changes. I also have a very solid test suite which is helping the coding agent a lot
Opus 5 doesn't really even speak coherent English. I'm not sure what's going on, but it can't explain anything. It still does an excellent job with code and writing tests and code review and creating and completing a plan, and it seems to be able to understand English instructions, but it sure as hell can't explain what it did or how to use the code it wrote.
That was true before they announced the watermarking, I'd already started to back off of using Opus as much because I like to understand what the model is doing and have it write documentation I can use to reproduce its results, but maybe watermarking was already in there unannounced.
I think it could be the watermarking, but at this point they might be deliberately complicating the prose so that we ask clarifying questions and that leads to more token spend.
I bounce between Sol high/medium and Luna max. I don't know why you'd use anything between Luna max and Sol medium. Luna is so extremely cheap and cranked up to max it does anything I'd want Terra to do for a fraction of the cost. What is Terra for?
One thing I've really noticed with Luna Max is its speed. I've got a review script setup on a custom Pi extension. Luna finds some issues/some false positives, while Sol finds issues but disregards false positives. The biggest thing is Sol finishes in about half the time.
In my experience, Superpowers has begun to massively slow down the capable models at this point. The skills they add are incredibly bloated and just get you worse results nowadays, tbh.
I am seeing essentially deterministic results with Terra running a custom browser automation agent across >100 interaction events.
The harness is everything. If I just threw something like Codex at this and said "good luck" I wouldn't make it beyond 5-10 interactions. I tried that already. Carefully designing the views and tools over the environment is where you can go from 50% to 99.9999%.
Hand-crafted adapters that sit between playwright primitives and the agent loop are the secret sauce. The goal is to insulate the agent from the raw DOM without any loss in fidelity regarding the logical business information and available actions.
+1 I have found extremely reliable systems require a mix of deterministic "adapters" is a good word for agents to actually get through the workflows I've created. I'm still amazed that it can work with both those and some "intuition" to bend the rules around the adapters if prompted.
