a tool that actually shows what third party SDKs in your own apps are phoning home to would be huge, most people have no idea what their own dependencies are sending out. everyone builds network level tools but that app level visibility is missing.
Oh, now this is important. I'm not too familiar with app security, especially telemetry, but I will look into the process of making something like this.
most of the pain at scale isn't the agents themselves, it's observability. once you're past a handful you basically need per-agent tracing or you're debugging blind, worth borrowing tooling from how people do this for microservices
Depends on your setup but we've had success with building custom harnesses and processing metrics via OpenTelemetry + Prometheus. This is running on a massive K8s cluster spinning up pods per N agents.
If you are using a proprietary harness like CC or Codex then look into automatically ingesting their memory logs and internal datastores to track progress and then instrument the host to send back metrics
Depends a lot on what kind of prose. For anything that needs to sustain a consistent voice over a long piece, the models that let you steer style with a short style guide up front tend to do better than ones you have to constantly re-prompt mid-way through.
reply