CompressDog

Two Compression Claims That Failed a 320-Image Test

Our page promised "about 3% smaller" and "5-10% smaller". Re-run on 320 images with SSIMULACRA2, the first is 7% and better, the second was the wrong comparison.

2026-10-22 ยท 6 min read

Open our compression page. Under the row of small buttons there are captions with numbers in them. "About 3% smaller in our tests." "+0.6%." "About 2% larger." "5-10% smaller than the default."

Those captions exist so you can pick a setting without guessing. They came from six images and a PSNR score. Fixing them meant testing the rest of the quality curve as well, which ended up changing the default. If the technical words in the table below are unfamiliar, this explains the slider first.

Six images is a spot check, not a test. And PSNR was the wrong metric. Our own encoder code says so, which is why it was replaced with SSIMULACRA2 in the first place. So the captions were quoting a number from a method we had already stopped trusting.

This is the re-run. 320 images, SSIMULACRA2, and the same parameter builders the site actually uses.

Two captions did not survive. One of them was wrong in the direction that made our default setting look better than it is.

What we ran

Same setup as our chroma subsampling test, so the two sets of numbers can be compared directly.

320 images, at most six from each folder of the collection, so no single style dominates. Each one is scaled so its longest side is 768 px and saved as a PNG. That PNG is the reference everything is scored against.

Then the same reference is encoded once per setting, at quality 90, with chroma subsampling held at 4:4:4 throughout. One thing changes between runs: the setting.

The settings are not invented for this article. They are imported from src/engine/encode-options.js, the file the site itself runs. Copying the parameters into a separate test script would have measured the test script.

Each output is scored against the reference with SSIMULACRA2. Higher is better. As a rough guide, below 70 usually shows visible damage and above 85 usually looks like the original at normal viewing size.

The numbers

Caption one: "about 3% smaller"

The standard-tables setting is not 3% smaller. It is 7.28% smaller, and its mean score is slightly higher than the default rather than equal. On 306 of the 320 images the score went up.

That is the part we did not expect. Smaller files that measure better is not supposed to come out of swapping a quantisation table.

What we think is happening: "quality 90" does not mean the same thing for the two tables. Mozjpeg's tuned table is built to hold up when images are squeezed hard, and at 90 it uses its budget differently. We have not traced which coefficients change. That is a guess, not a finding, and we are labelling it as one.

The "no visible difference" half of the old caption was fair, incidentally. 86.74 against 86.12 is not a difference anyone will see side by side. It was the size number that was wrong.

Caption two: "5-10% smaller than the default"

This one is not a small error. It compares the wrong two things.

At the same quality number, slider at 90, pick jpegli, the file comes out 0.36% larger. What you get for those extra bytes is a much better score: 89.60 against 86.12, and jpegli won on all 320 images.

The "5-10% smaller" figure comes from a different comparison: equal measured quality, not equal slider position. We keep a calibration table for exactly that, and at matched scores jpegli does come out smaller. Over roughly 5-10%.

Both statements can be true at once. But the button sits next to a quality slider, so "at the same setting" reads as "at the same slider position". Under that reading, the caption is wrong.

What we changed

Three captions, in both languages:

The trellis caption held up better. It promised +0.6% and measured +0.44%. But it also promised "a little more detail", and that only happened on 195 of the 320 images. On the rest it made no difference or scored lower. The wording now says that.

The change we are not making yet

If standard tables are smaller and score better at quality 90, why does the tuned table stay the default?

Because we tested one quality. The tuned table exists for a reason, and the reason has to do with how images hold up when you squeeze them hard: quality 50, 60, 40. Nothing in this test speaks to that range.

Switching the default on a single data point would be the same mistake as the captions, one level up. So it stays, and the next test is the low-quality end.

Caveats, in the order they matter

The images are 768 px. Most photos people compress are bigger, and encoders do not behave identically across scales.

Everything was measured at quality 90. Nothing here says anything about 60.

The sources are mostly already-compressed JPEGs. That is realistic, but it means we are measuring the cost of a second pass, not a first one.

One collection, one build of each encoder. Mozjpeg and jpegli both ship changes.

And a score is not your eyes. SSIMULACRA2 tracks human judgement well on average, which is why we use it. But "on average" is carrying weight in that sentence.

Reproducing it

The pipeline is short enough to rebuild in an afternoon. Resize each source so its longest side is 768 px and save a PNG, which becomes the reference. Encode that same reference once per setting at quality 90, then score every output against it with SSIMULACRA2. Pillow handles the resize; the scorer is a single binary called once per file. It took about eight minutes for 320 images on a laptop, and most of that was the scorer rather than the encoders.

If you run it on your own library and get different numbers, we would like to hear about it. The address is on the about page.

Open the compressor

Related reading