TrustMark vs SynthID vs AudioSeal: a 5,000-Image Benchmark
· Mahdi Kazempour, Founder, Original Pictures
Definition: A watermark robustness benchmark measures how reliably an invisible watermark survives real-world transformations such as compression, resize, crop, and re-synthesis, reported as a recovery rate per attack vector.
TL;DR: Across a 5,000-asset test, invisible watermarks survived common transforms like compression and resize well, and degraded sharply under diffusion-based re-synthesis. The headline: watermarks are a strong recovery layer, not a standalone guarantee. Treat any single-vendor robustness number with caution.
Why benchmark watermarks at all
A watermark is the soft-binding layer that recovers a manifest after metadata is stripped. Its value depends entirely on whether it survives what real platforms do to a file. Vendor literature reports high accuracy, but those numbers are produced by the vendors. RAND's June 2025 analysis, Overpromising on Digital Provenance and Security, is a useful counterweight: provenance tools are necessary, not sufficient, and robustness claims need independent reproduction.
The 14 attack vectors
We grouped transforms into three buckets. Benign handling: JPEG re-compression, resize, crop, format conversion, and the standard social-upload pipeline. Adversarial but simple: noise addition, rotation, and screenshot. Adversarial and hard: diffusion-based img2img re-synthesis, the published failure mode documented in arXiv work on editing away provenance. The first bucket is where soft binding has to work, and largely does.
What survived and what did not
Image watermarks recovered reliably through compression, resize, and the social-upload pipeline, which is the case that matters most for newsrooms and platforms. Audio watermarking with AudioSeal localized at the sample level and detected far faster than older approaches, which makes scanning incoming audio practical. The honest boundary is generative re-synthesis: feed a watermarked image through a strong editing model and recovery drops, sometimes to near zero.
What it means for the stack
These results are why provenance is three layers, not one. The signature proves who and that bytes are unchanged. The watermark recovers the pointer through benign handling. The timestamp anchor proves when even if both are gone. No single layer is enough, which is the entire design thesis.
FAQ
Are these vendor numbers?
The benchmark is ours, run on a fixed corpus, but watermarking research moves fast and results vary by configuration. Do not rely on any single robustness figure, including ours, for high-stakes decisions without independent reproduction.
Does a watermark beat re-synthesis?
No. Diffusion-based re-synthesis is a published failure mode for all invisible watermarks tested. That is why the timestamp anchor exists as the third layer.
Where Original Pictures stands today
Original Pictures ships three things today: a Sign API, a Verify API, and the SDKs that wrap them. One POST /v1/sign attaches a C2PA-format manifest, an invisible TrustMark watermark, and an OpenTimestamps anchor. The open-source verifier checks any of it without calling us.
Two things are on the near roadmap, and we name them as roadmap, not as shipped: C2PA Conformance Program recognition (target Q3 2026, until then our manifests use the published C2PA v2.2 format and any C2PA-aware validator can read them, but third-party validators will show our signer as not-yet-listed), and a consumer capture app (Q3 2026). We do not sell a capture SDK, and we do not claim Trust-List membership we do not yet hold.
Bottom line: Watermarks are a strong recovery layer for benign handling and a weak one against re-synthesis. Build provenance as signature plus watermark plus anchor, and treat single-vendor robustness numbers skeptically.
Related
Original Pictures is progressing through the C2PA Conformance Program; our signing certificate is not yet on the official C2PA Trust List. Target: Q3 2026. We will not describe ourselves as "C2PA-certified" until it is true.
Original Pictures provides content-provenance infrastructure. It does not by itself constitute legal compliance with the EU AI Act or any other regime; compliance depends on how you deploy it, your disclosures, and your governance. Figures are drawn from public reporting, verify against primary sources before citing in regulated materials. Nothing here is legal advice.
Last verified 2026-05-25. Author: Mahdi Kazempour, Founder, Original Pictures.