WARC itself does have a method of dedup if the response is the same (or mostly the same) but terrible if content changes.
WARC itself does have a method of dedup if the response is the same (or mostly the same) but terrible if content changes.