Support for Google Zopfli and Brotli compression

[H4ndy]

I would like to see at least Zopfli-Support, which is a DEFLATE-compatible open compression algorithm which is slow but has better compression.

Reference implementations are available and details can be found here:
https://github.com/google/zopfli
https://en.wikipedia.org/wiki/Zopfli

Brotli is NOT deflate-compatible but has even better ratios (beating LZMA2) and is intended to replace gzip on the web for data transfers.
http://google-opensource.blogspot.de/2015/09/introducing-brotli-new-compression.html
https://github.com/google/brotli/
http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf

Cheers!

[spwolf]

Thanks Manuel!

are there any test results comparing speed with kzip or 7zip deflate?

[H4ndy]

There’s a comparision table for the “Canterbury corpus” in http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf at Page 4 and the following pages with “real world” data.

[spwolf]

@H4ndy:

There’s a comparision table for the “Canterbury corpus” in http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf at Page 4 and the following pages with “real world” data.

hm, first test shows zoplfi being 70x slower to compress than deflate for 0.2% better compression ratio… this is likely because focus of their deflate is strong er compression for png images.

However in archiver, this would be very impractical since you can use 7z format for instance to achieve much better compression at much faster speed. For instance they show 7z at ultra setting being 20x faster than zoplfi while getting much better compression.

Similar implementations of stronger but much slower deflate existed before (kzip), and are used for various png re-compress tools.

[H4ndy]

Fair point. Zopfli is already in use for PNG re-compression.
Since it it deflate-compatible no further support is needed on PA side I understand.

But if some sort of “real” format of Brotli gets adopted at least decompression support for that could be implemented (like PA already supports some more exotic formats).

Thanks for your considerations so far

[spwolf]

@H4ndy:

Fair point. Zopfli is already in use for PNG re-compression.
Since it it deflate-compatible no further support is needed on PA side I understand.

But if some sort of “real” format of Brotli gets adopted at least decompression support for that could be implemented (like PA already supports some more exotic formats).

Thanks for your considerations so far

if they add it for 7z or zip, we will use it of course, but again, this seems to be made for web page compression where speed of extract is important… test shows LZMA being 8x faster at ultra than Brotli with LZMA being tested at low window sizes, which means comparing it to classic LZMA window sizes, it would get much better results.

For them though, most important part is again speed of decompression, which is only where it is likely good.

So I dont think you will see this implemented in real archivers, unless someone makes modifications that make it much better.

Now on the other side, zpaq…

[spwolf]

@H4ndy interesting… since .pa is coming, we just tested brotli vs zstandard:

43426928  0.921s:  bro.exe --verbose --force --input enwik8 --output 1 --window 27 --quality 0
42456665  1.139s:  bro.exe --verbose --force --input enwik8 --output 1 --window 27 --quality 1
36950376  2.371s:  bro.exe --verbose --force --input enwik8 --output 1 --window 27 --quality 2
36685096  2.839s:  bro.exe --verbose --force --input enwik8 --output 1 --window 27 --quality 3
35623532  4.446s:  bro.exe --verbose --force --input enwik8 --output 1 --window 27 --quality 4
33409515  8.269s:  bro.exe --verbose --force --input enwik8 --output 1 --window 27 --quality 5
32447946 11.654s:  bro.exe --verbose --force --input enwik8 --output 1 --window 27 --quality 6
31059249 21.435s:  bro.exe --verbose --force --input enwik8 --output 1 --window 27 --quality 7
30327408 35.272s:  bro.exe --verbose --force --input enwik8 --output 1 --window 27 --quality 8
29686256 61.574s:  bro.exe --verbose --force --input enwik8 --output 1 --window 27 --quality 9

40859654  0.905s:  zstd.exe c enwik8 0  1 52 1 27
37647556  1.124s:  zstd.exe c enwik8 0  1 52 2 27
35643321  1.358s:  zstd.exe c enwik8 0  1 52 3 27
34878618  1.544s:  zstd.exe c enwik8 0  1 52 4 27
34801006  2.246s:  zstd.exe c enwik8 0  1 52 5 27
33269248  3.042s:  zstd.exe c enwik8 0  1 52 6 27
32588280  3.931s:  zstd.exe c enwik8 0  1 52 7 27
31796598  5.460s:  zstd.exe c enwik8 0  1 52 8 27
31293605  7.363s:  zstd.exe c enwik8 0  1 52 9 27
27259613 62.759s:  zstd.exe c enwik8 0  1 52 19 27

29686256 61.574s:  bro.exe --verbose --force --input enwik8 --output 1 --window 27 --quality 9
27259613 62.759s:  zstd.exe c enwik8 0  1 52 19 27

29686256 61.808s:  bro.exe --verbose --force --input enwik8 --output 1 --window 25 --quality 9
27287874 62.463s:  zstd.exe c enwik8 0  1 52 19 25

so at same speed it has lower compression in testing by @eugene , and in my testing with same compression, it is 2x slower… so it seems like inferior to zstandard.

[H4ndy]

@spwolf Yeah it really is only for high-demand resources where you want to squeeze the smallest possible size out of it and you do not care that it takes half an hour to compress it in the first place. It’s bandwidth optimization.

[spwolf]

@H4ndy I know that’s the official line, i hope it works for them :).

[H4ndy]

@spwolf Probably once you start serving some resources a few million times a day
Anyway, thanks for still checking it out

[spwolf]

@H4ndy yeah… pretty much everything was tried… same goes for apple’s port of zstd. Not efficient for archiver at all.