I created a tool for this: https://github.com/Use-Tusk/fence

Same thoughts - I wanted a "permission manager" that defines a set of policies agnostic to coding agents. It also comes with "monitor mode" that shows operations blocked, but not quite an audit log yet though.