What features do you want in new format?

[spwolf]

Tell us what features you want from new format…

[spwolf]

Strong compression? Fast compression? PDF compression? Encryption? Multivolume? PAR-like recovery???

[amh209]

PDF compression would be nice. I send a lot of PDF files, so I would benefit from that one.

[TBGBe]

Are you developing a new compression format OR looking for a new format to be supported in PA :confused:
@spwolf:

Strong compression? Fast compression? PDF compression?

Yes, please. :D
@spwolf:

PAR-like recovery???

Well definitely some form of error recovery. :cool:
@spwolf:

Encryption?

Not so necessary, can use PAE for encryption.
@spwolf:

Multivolume?

Not sure about this. Maybe multiple stand-alone archives based on archive size - but without splitting contents across archives, sort of “best fit to CD/DVD size”.

Of course, you omitted self extracting and media file optimisation :p

[spwolf]

do I always have to do all the work? :-).

So what would Your perfect format have in terms of features Terry?

[Luxor]

In order of preference.

Fast compression.
PDF compression.
PAR-like recovery.
Strong compression.

But I’m easy, I can wait for them all. :p

[spwolf]

But dont be limited to what I say… JPEG, PNG, GIF, XML, MP3, etc compression?

[TBGBe]

@spwolf:

So what would Your perfect format have in terms of features Terry?

I am biased towards Backup function so:-
Good loss-less compression
Media file compression (minimally video and picture)
Error Recovery
“Intelligent” Multi-volume
No less than O.S. file size limits on source or archive (quantity or size)
Ability to extract single/multiple files (so no need to extract whole archive as with solid).

Doesn’t have to be super fast, but not slower than Zipx maximum at worst.

For general (non-backup) use, then some extras:-
Create Self Extracting archives
Update existing archive (add/freshen)
Strong Encryption

[spwolf]

what about proper versioning, and byte level differential backups, where literally only differences are stored?

[davidsplash]

how about a repair tool for all archivers and sfx files

[Socrates]

My preferences (if doable):
Better PDF compression
Differential backups

[TBGBe]

@spwolf:

what about proper versioning, and byte level differential backups, where literally only differences are stored?

That only works with good error correction :p
So version 2.0 :D

[spwolf]

@TBGBe:

That only works with good error correction :p
So version 2.0 :D

well that part is already almost done :)

[spwolf]

@davidsplash:

how about a repair tool for all archivers and sfx files

read my post david and then read yours… what does repair tool for “all archives” have to do with brand new format in PowerArchiver?

[spwolf]

@Socrates:

My preferences (if doable):
Better PDF compression
Differential backups

yes, but do not be limited to only things I mentioned… what other things would You want there?

[spwolf]

here is the list of possibles for special compression codecs:

jpg
mp3
pdf/ps
avi
mp4/mpeg/mkv
docx/xlsx/pptx
flv/swf
vob/m2ts/ts
zip
cab
setup.exe/data1.cab
html/htm/xml/py/php/css/js
psd
png
gif
bmp
wav
tar/tar.gz/tgz/tar.bz2
exe/dll/obj/lib
rar
c/cpp/h/hpp/cs/pas
hlp/chm
torrent
msi+B20
mdf
dts
iso

All of these require some special form of compression and each will take a lot of time… so better voice your opinions because if you do not, it might be a lot of time before your “wish” is taken into consideration…

[davidsplash]

in response to my anaswer to my last post what i was trying to say we dont want to see new formats that cannot be repaired. should have made my post clearer but when using android mobile not easy.

[davidsplash]

one format i think we would like to see is sit and sitx decompression.

one i would like to see is .afa
A format that compresses and doubly encrypt the data (AES256 and CAS256) avoiding brute force attacks, also hide files in a AFA file. It has two ways to safeguard data integrity and subsequent repair of the file if has an error (repair with AstroA2P (online) or Astrotite (offline)). The speed of compression / decompression is superior to other systems by making extensive use of multiple processors

one i do like a lot and use is jar the successor to ARJ really good compression rates here

[spwolf]

I dont think David reads our posts too much :)

[TBGBe]

@spwolf:

better voice your opinions because if you do not, it might be a lot of time before your “wish” is taken into consideration…

Maybe you should open a poll?

Anyway Yes, please - :D
jpg
avi
mp4/mpeg/mkv
flv/swf
png
gif
bmp
wav
exe/dll/obj/lib

Lower priority - :cool:
mp3
pdf/ps
docx/xlsx/pptx
html/htm/xml/py/php/css/js
c/cpp/h/hpp/cs/pas
hlp/chm

Meh - :p
psd
torrent
mdf
dts
msi+B20
vob/m2ts/ts
iso

Not sure what you mean with these existing formats (conversion?)-
zip
cab
setup.exe/data1.cab
tar/tar.gz/tgz/tar.bz2
rar

[spwolf]

like this? :)
http://powerarchiver.idea.informer.com/proj/?show=&show_all=1

[TBGBe]

Yes. I like that.
Some of them could do with better description though.

E.G. Just what is meant by <data segmentation="" and="" reordering="">in terms of a compression format :confused:</data>

[spwolf]

i will let eugene explain that… you can also vote, add and comment there!

[spwolf]

one problem there is that anyone can do multiple votes, meaning dont rate everything for the sake of rating, just rates one you feel really passionated about!

[spwolf]

thing was, we didnt want you have to login to vote

[TBGBe]

Err, who is “eugene”?

Note to self - must show spwolf the “Edit” button sometime!

[spwolf]

i hate editing as you wont get email notification with edited parts :).

Eugene is our compression engineer, working on actual format (same guy did patchbeam engine, rar speed improvements, wz jpeg decompression, etc).

[davidsplash]

i was just pointing out features in existing formats that would be nice to have included in pa. why right a new format unless you want a format that includes everything.

I apologise its very difficult using android and your forumn nearly impossible

[mwb1100]

@spwolf:

same guy did patchbeam engine

You guys did patchbeam in-house?

Nice. For some reason I had assumed it was technology bought/licensed from a 3rd party.

[spwolf]

@mwb1100:

You guys did patchbeam in-house?

Nice. For some reason I had assumed it was technology bought/licensed from a 3rd party.

all ours…

[spwolf]

@TBGBe:

Not sure what you mean with these existing formats (conversion?)-
zip
cab
setup.exe/data1.cab
tar/tar.gz/tgz/tar.bz2
rar

re-compression… if you compress them as they are, you will gain nothing… if you extract and compress them, between 15 and 50% gain in compression is possible.

[Mameluke]

I would most likely want to see the following in this order if possible please :)

1. Internal and External (RAR style) recovery record (with choice of percentage just like RAR).
2. JPEG compression (got many pictures that are quite large)
3. Strong Encryption with ability to encrypt file names (Like Winrar and 7z)

Thank you :D

Mame

[davidsplash]

would love to see a format that actually can compress media files better than we have at present. At the moment most media formats are donme by several formats i would like one format to do them all e.g wave mp3 and jepg format compression all in one archive format at present i have to use several formats in order to get the best comopression for each media format like sound or images

[davidsplash]

a format that preserves data like who made it and what version of the archive format made it. usefull for people who need this info

[By-Tor]

I’d still like to see wz jpeg compression for zipx.

[spwolf]

@By-Tor:

I’d still like to see wz jpeg compression for zipx.

it is still eventual possibility… however, while we researched jpeg compression last year, our first version was slightly better compressing than WZ Jpeg and 3x faster on dual core machine… and that could be done even better with 20% better compression than wz and up to 3x faster speed on dual core machines :).

so which one would you want? :)

[By-Tor]

@spwolf:

it is still eventual possibility… however, while we researched jpeg compression last year, our first version was slightly better compressing than WZ Jpeg and 3x faster on dual core machine… and that could be done even better with 20% better compression than wz and up to 3x faster speed on dual core machines :).

so which one would you want? :)

Why not both? :p

[spwolf]

it takes 2 months for engineer to implement each, so during those 2 months he could do:PDF, MP3, PNG instead etc, etc.

it is very time consuming and demanding project, which is why nobody but PA can read these archives currently.

[By-Tor]

Ah. From the sound of it, I thought the implementations were close to complete. I guess the first thing I’d like to see then is the wz jpeg compression, just for compatibility. I’m less concerned with a unique format right now as portability issues would be a hindrance. For another unique archive format to catch on, I think you’d need to release an open source portable command line version at the very least before it would be truly useful among other users.

[spwolf]

@By-Tor:

Ah. From the sound of it, I thought the implementations were close to complete. I guess the first thing I’d like to see then is the wz jpeg compression, just for compatibility. I’m less concerned with a unique format right now as portability issues would be a hindrance. For another unique archive format to catch on, I think you’d need to release an open source portable command line version at the very least before it would be truly useful among other users.

compatibility with only WinZip though, no other utility, and certainly no free or open source utility… so what is the difference really between PAF, IZEHRBLAH, and ZIPX, when only two products support extracting ZIPX archives.

Problem is that ZIP itself, due to old nature of format, will never be anywhere as good as modern format, and whatever extensions are added to ZIPX not only that they cant be as good as new format, they will also not be compatible with other products either.

Who knows, maybe PAF will be open format and maybe there will be free extraction tools, eventually anyway :)

[By-Tor]

WinZip has a large userbase, regardless of whether we like it or not. 7-zip can handle most of ZIPX, with the exception of jpeg compression, but, Igor has stated he would implement some form of it if he saw a need.

ZIP by its nature is just a container format. So, you’re right, it doesn’t matter much what you call it. I’m just thinking about trying to capture the largest audience possible.

I’m by no means anti-PAF. heh. I’d love to see a better format. After seeing so many archive formats come and go, I know it takes time for any format to catch on if at all.

[spwolf]

@By-Tor:

WinZip has a large userbase, regardless of whether we like it or not. 7-zip can handle most of ZIPX, with the exception of jpeg compression, but, Igor has stated he would implement some form of it if he saw a need.

ZIP by its nature is just a container format. So, you’re right, it doesn’t matter much what you call it. I’m just thinking about trying to capture the largest audience possible.

I’m by no means anti-PAF. heh. I’d love to see a better format. After seeing so many archive formats come and go, I know it takes time for any format to catch on if at all.

yeah, but there are much bigger differences to be made with new, proper format… you cant just put something into zip and get good results, it doesnt work that way.

compress folder with zipx lzma and with 7zip lzma, and notice that the difference in size could be up to 50%, simply because zip doesnt and will not ever have proper solid compression.

point with paf is that we can make something unique with it that nobody else has - compression for many popular formats… weather it becomes next biggest thing is something else, we plan to use it in our next backup utility as well, where nobody does any of these things and space savings will be significant.

[spwolf]

but pretty much, everything on that list takes at least a month to implement for single engineer… thats why nobody has done it before (expensive) and why all the new formats are very much alike (variations of old).

[By-Tor]

@spwolf:

point with paf is that we can make something unique with it that nobody else has - compression for many popular formats… weather it becomes next biggest thing is something else, we plan to use it in our next backup utility as well, where nobody does any of these things and space savings will be significant.

I totally agree. I wasn’t clear on how far reaching you wanted things to be.

[spwolf]

@By-Tor:

I totally agree. I wasn’t clear on how far reaching you wanted things to be.

it is massive undertaking though, and it takes a while…

but it will be worth it for instance for backups, where we will be able to do 50% more efficient backups that rest of the current utilities :)

[spwolf]

keep in mind we have been “researching” this for past 2 years, so we are not staring from 0 here…

things we did so far:

• ver 1 jpeg compression - 3x speed of wz jpeg with slightly better compression. Aim in PAF: maintain similar super speed while adding extra 20% compression compared to WZ Jpeg.
• ver 1 differential and versioning system - compared to leading Mozy backup service, we had 30% smaller differential backups (which means 30% faster, 30% more effcient, 30% less costly for bandwith and storage space), without strong compression implemented… Goal is to tie it together into PAF and enable stronger compression that would give us 50% gain total.
• Various multicore research into improving speed for other operations during compression, not just compression codec, that should add extra 20-30% speed improvements over current formats (with similar codecs).
• modification of lzma2 codec into pa-lzma, to fit our format better, and better overall performance (to be released as open source), this is 80% done as it is…

[spwolf]

New formats idea site:
http://ideas.powerarchiver.com/

[Socrates]

The sample size on your survey is so small, I would caution against taking it too seriously.

If you found a way to ask a broader audience what they want, I would be shocked if multi-volume support would be in the top five.

There may not be a better way to reach a broader audience, and, if so, having established voting, you probably have to act based on the suggestions you received. Still, I fear that in so doing you be will be spending a lot of effort on a feature that really won’t appeal to that many people. I suspect 5-7 of the other options would be more appealing, even if they are not ones I would use.

So if you try this again, you might explore ways of getting input from a larger sample.

[spwolf]

no need to worry, while ideas site is interesting we still have our own schedule and goals to work with (which is getting better compression on things that are not compressible currently).

multi volume feature is really simple to implement, but we probably would not do it that way if we didnt get enough votes.

[Socrates]

If it is simple, then that’s a compelling reason to do it.

What I often don’t know is how difficult it is to implement a new feature. Some might seem easily, but be difficult (or next to impossible) while others that seem difficult might be a snap.

[spwolf]

@Socrates:

If it is simple, then that’s a compelling reason to do it.

What I often don’t know is how difficult it is to implement a new feature. Some might seem easily, but be difficult (or next to impossible) while others that seem difficult might be a snap.

hardest things on that lists are new codecs for pdf, jpeg, mp3… thats both hard and time consuming. Everything else on that list will probably take less time all together (!) then building special jpeg compressor.

[Socrates]

I suppose I would have thought special compression would be especially difficult. But not that much more difficult.

Thanks for letting us know.

At some point (and if not too complicated), could you explain the relative difficulties of building new compression from pds versus jpgs. I assume the former would be easier (more white space). Perhaps, though, that assumption is borne of ignorance.

[spwolf]

#1 goal for the format: special codecs for file types that are most used currently yet can not be compressed by current tools.

basically this is compression for complicated (already compressed) file types.

so what you have to do is take file apart, and divide it into parts that can and that can not be compressed (which is why jpeg, png, docx, pdf that is compressed, can not be compressed further usually)… then compress the part that can be compressed with special codec designed for that format. All of this is done transparently to the user of course, and usually quite fast if done right.

But for each format (mp3, jpeg, png, pdf, docx, odt, etc), special codec is required. So there is a lot of development work to be done here. There are also no examples of such work, and only very few utilities do it - for instance, Stuffit has a lot of special compressors, but it is available only if you give out your credit card (no actual free trial), while most other utilities do not have anything but general codecs.

Reality is that most people compress things that are already compressed, so using zip, or rar or 7zip on most things people usually backup or send via email will not result in great savings, or sometimes savings at all. If you compress jpegs to send over email or upload somewhere or simply backup, you will not gain any compression. On the other hand, with special jpeg codec, you can expect 20%-30% gain on your full album of pictures.

For instance zip has one most common codec which is deflate. WinRar has single codec too.
Now .paf/pa/power will have 5-6 at least within next 2 years. So you can imagine how big the task is. But the gains are big too so it is worth it.

[spwolf]

difference between pdf and jpeg is that while for both, you have to develop special recompression algorithm, pdf uses deflate compression from zip and then its contents once opened up, can be compressed well with general codecs.

on the other hand, jpeg has many variations so first recompression has to be able to take apart various jpegs and then you have to build completely custom codec to compress that picture inside.

So basically it is double the work compared to PDF.

Advantage with PDF is that we can use a lot of the code to recompress other formats like PNG, DOCX, ODT, SWF, since they all use deflate to give them (weak) compression.

For instance, DOCX containts XML files inside that are compressed with deflate. However since they are compressed, if you try to compress them again, you will gain very little… but if you unravel weak compression and then apply stronger one, big gains are possible.

Here is test example.

1. Contract in DOCX format - 104 KB
2. DOCX compressed with 7zip - 95KB
3. DOCX recompressed properly - 64 KB

So thats 40% gain on DOCX file for instance. Imagine if you have many of them on your computer, or if your company sends many via email or backup service… Time and cost savings are quite significant here.

Now actual % gained is different for different formats and there are further optimizations possible (for instance detect pictures and text inside single file differently and compress them with their own codecs), but you can see how much potential this has.

Main thing here is that it has to be done seamlessly and it has to be fast, otherwise people will not use it. And then we come to #2 part of new format - multicore optimizations.

[Socrates]

thanks!

[RJWaring]

I would have to say that Jpeg, and PDF Compression is a good idea! as your possibly aware I have brought Picture compression up a few times in the past with a good warrent for it.

But with a list aslong as the above there are so many other good things to choose from.

I have made some votes. :o)

[davidsplash]

i would like the new format to introduce some sort of protection so that if the archive is changed it tells you by whom and can stop this if you dont wnat it updated but would like people to extract files from it only

[davidsplash]

One idea would be to have time limited archives that actually destroy themselves after a set amount of time. saw this in america well worth it if you are sending things over the internet. They also had archives that only let you open them a set number of times.

[davidsplash]

to sound a downer you can not have every feature in the new format as that would slow it down.

What we would like to see is several new archive formats .

For instance pdf compression

wave compresion

etc

would love to see all these new formats use the plugin feature in pa.

[FiatLux]

The option that you when in the “with options” interface then may choose both Encrypt and make SFX in one operation, and implemented using a technique that can handle both many andd/or large files….