Poor compression of >20GB exe/msi/cab sample
-
Thank you for sticking around, we’ll get to them bottom of this.
Please see the step-by-step instructions I’ve posted here for configuring the compression string manually:
https://encode.ru/threads/130-PowerArchiver?p=53664&viewfull=1#post53664
It is easier to attach images to a post there, so please excuse me for having to redirect you there for instructions. There are precise instructions there though, so your detour is worth it.
For ZIPX, we use the BricolSoft Zip ActiveX control - this excellent library may be worth for you folks to check out as well. After all, nobody on this thread invented 7-Zip, and nobody here invented ZIPX either!
-
@diskzip said in Poor compression support:
https://encode.ru/threads/130-PowerArchiver?p=53664&viewfull=1#post53664
cool, will test it out with that setting sometimes soon.
-
We have just released DiskZIP 2018, with the following enhancements:
https://encode.ru/threads/2763-DiskZIP
Thank you very much for the feedback you provided on this thread regarding the usage of CPU cores. This issue is now addressed by way of utilizing all available CPU cores (with the exception of the highest compression profiles, which limit to two CPU cores in order to ensure the best compression ratios).
Please also note, you would now need to choose the second command line override string instead of the first, because in DiskZIP 2018, the first command line override string again forces BCJ2 compression :)
I know, ironic! The customer is always right, and that’s what they wanted…
So please excuse the inconvenience caused by the inaccurate instructions - as you will see for yourself when you take a look at the custom compression strings though, the second one does engage 7-Zip’s automatic file detection filters.
I would like to thank you all once again, on behalf of the entire DiskZIP team, for your feedback - and we look forward to any further feedback you may have on the 2018 release. Wishing you all a great weekend!
-
@diskzip now we get PR releases on our forums, do you have a link to it so i can delete the huge post relevant mostly to your product?
-
@spwolf You are right, please excuse my enthusiasm. I have made the edit directly, please don’t hesitate to let me know if it is not satisfactory.
-
@spwolf Have you had a chance to test our native compression yet? Please note, as I described in my new release announcement post, to use an auto-switching filter, you now how to use the second command line override parameter (instead of the first as it was in the prior version). Thanks!
-
@diskzip whats native compression, 7z?
Did not have a chance to try new version yet, i might this week.
As to the PA format, our current record with simple/optimized setting is:
PA 17.01.03: 2,460,182 kB (d2G, mt1)
7zip max settings: 2,593,514 kB
Your sample: 2,600,325 kBWe adjusted extension groupings, added more extensions to the exe/dll group that our exe filters are applied to and added unlimited size blocks to get better compression on samples over 10GB.
I think we can do at least 20-30M better with current settings, not everything is still perfectly grouped together for this sample.
So currently around 130M better than best 7z settings, 136M better than dzip.
-
@spwolf said in Poor compression support:
@diskzip whats native compression, 7z?
Did not have a chance to try new version yet, i might this week.
As to the PA format, our current record with simple/optimized setting is:
PA 17.01.03: 2,460,182 kB (d2G, mt1)
7zip max settings: 2,593,514 kB
Your sample: 2,600,325 kBWe adjusted extension groupings, added more extensions to the exe/dll group that our exe filters are applied to and added unlimited size blocks to get better compression on samples over 10GB.
I think we can do at least 20-30M better with current settings, not everything is still perfectly grouped together for this sample.
So currently around 130M better than best 7z settings, 136M better than dzip.
By native compression, I was asking whether you were able to reproduce our results on our testing data set using DiskZIP and the updated compression string for the 7ZIP file type, natively inside DiskZIP, without using 7z.exe or 7z.dll or any other external process.
So can you share a compression string - or other instructions to configure PA - so that I may be able to reproduce your results internally?
Also, glad to see you were able to push 7-Zip 6 MB more than the built-in DiskZIP. What compression string did you use to achieve this - or what version of 7-Zip?
We should be able to replicate the exact same result in DiskZIP, of course; its just a matter of discovering the compression string.
Thanks again, and congratulations!
-
@diskzip yes, i was… when it comes to 7z, I just maximized settings in 7z gui and added same switches you use for sorting, and that gave me 6-7 MB smaller file. Try it out and see what are the differences between your internal switches.
There is no overly complicated string or anything like that in PA, just select 2000 for dictionary and mt1, extreme setting of course. Same as before.
Update should be out today or tomorrow, depending on QA team testing, there quite a number of things changed for other parts of PA. Update from last month already got the same settings down to 2.599… There is no change in codecs.
We should get few mb more at least in the end. It was useful for us as a testing in the end, to add more exe/dlls formats on the list, as well as sorting/big block switch for files over 10G total.
-
Of course. Glad I could help :)
-
New record with Razor: 2,413,444 kB
https://encode.ru/threads/130-PowerArchiver?p=54599&viewfull=1#post54599 -
@nikkho said in Poor compression of >20GB exe/msi/cab sample:
New record with Razor: 2,413,444 kB
https://encode.ru/threads/130-PowerArchiver?p=54599&viewfull=1#post54599Yes, while we have been bickering here about DLLs and EXEs vs native implementations; someone has actually improved something :)