does anyone have any recommendations for stronger multi-agent observability. I've not really seen anything outstanding in this space yet

Depends on your setup but we've had success with building custom harnesses and processing metrics via OpenTelemetry + Prometheus. This is running on a massive K8s cluster spinning up pods per N agents.

If you are using a proprietary harness like CC or Codex then look into automatically ingesting their memory logs and internal datastores to track progress and then instrument the host to send back metrics

does this mean you can disconnect from the internet entirely with the agent loop still running?

yes this is central to our thesis :)

is this an LLM?

this sounds so expensive, is this latency win worth picking worse models to make it affordable?

probably the only way to make something like this affordable. But the way models are trained right now is completely wrong for this anyway. They’re nowhere near good enough at estimating their own uncertainty and it’s RLHF’s fault for these ‘crutch plans’ we get. I guess the point of the architecture is that it shouldn’t need big smart models to work well, but whatever models you use, what’s the post-train for forked execution going to look like? This sounds so expensive to train too