Yes, only dirty (i.e. modified) pages profit from getting paged out to swap.
But while I don't have any hard data, I imagine that depending on the workload, there can be a punch of pages that are read during initialization, or a one-off workload, and never being read or freed up afterwards.
Just try activating a swap file, and see if it gets any usage, despite memory pressure being low.
(They obviously don’t page out, they are just discarded and read again when needed)