PA can take a long time "Preparing files to compress"

deipotent

I have noticed this before but never mentioned it until now.

If I select a folder containing lots of sub-folders and files, and then choose to create a ZIP archive of it using PA, the progress dialog appears with the info “Preparing files to compress. Please wait.”. It stays like this a long time before the compression actually begins. I assume PA is making a list of all files/folders (since the memory usage of PA gradually increases) and also getting the file sizes (to be used in calculating the progress).

Is it PA or the DynaZip library that builds this list ?

I ask because if it’s PA then I’d like to see this process speeded up (eg. maybe by writing a small library in C that can be called from PA). The same process in WinZip is much faster (ie. instead of memory usage increasing in 4K chunks, it jumps up in 500-800K chunks) and the actuall compression begins promptly.

Secondly, if you select the Cancel button while this file list is being built, the PA progress dialog says “Cancelling, please wait…” but the job is not cancelled immediately. Instead, you have to wait until the file list is completed (ie. indicated by the fact that the memory usage continues to rise gradually)…

So, to summarise I’d like to see the following improvements to PA:

Much improved speed when creating the list of files to compress! This is very important, because the long time PA takes to create this file list is probably PA’s greatest downfall. The actual compression speed is easily a match for WinZip, it’s just this preparation where PA takes a LOT LONGER.
Improved cancellation speed. It should actually interrupt the file list building process. If (1) is addressed, then this isn’t quite as important since PA will be in this “limbo” state for a lot less time.

Please can issue (1) be given the highest priority since currently PA cannot be used when compressing large numbers of files - The “Preparing to compress” just takes too long.

deipotent

FYI, a sample folder I was using to test had 2,627 sub-folders and 16,448 files in total.

I set it off 25 minutes ago and it’s still “Preparing to compress…”

WinZip started the actual compression after just 12 seconds of preparation!

spwolf

Hi,

Are you selecting single folder only? Try selecting that same folder and plus one extra file in the same root folder? Also does the root folder have many other subfolders and files?

For example:
C:\root folder\test folder

is the folder you are trying to compress. What happens if you compress:
C:\root folder\test
C:\root folder\test.txt

PA already does a fair bit of workarounds on DynaZip library, one of the most important ones and slowest ones is due to the “normal” path option being selected. Try de-selecting it and see if it works.

This is the same problem as mentioned with that other thing in support email you sent, PowerArchiver has to workaround DynaZip in several cases for correct path option to be recorded properly.

thanks,

deipotent

Thanks for the informative reply spwolf. Below are my answers (I disabled my Real-Time AV protection while performing the tests (see below))

@spwolf:

Are you selecting single folder only?

Yes.

@spwolf:

Try selecting that same folder and plus one extra file in the same root folder?

For example:
C:\root folder\test folder

is the folder you are trying to compress. What happens if you compress:
C:\root folder\test
C:\root folder\test.txt

Tried this but it didn’t help.

@spwolf:

Also does the root folder have many other subfolders and files?

The root folder had 63 files, and 18 other folders. In these 18 folders are 11798 files and 1431 folders/sub-folders. So I moved the folder I was compressing to a separate folder where no other files/folders resided, but that didn’t speed it up.

@spwolf:

PA already does a fair bit of workarounds on DynaZip library, one of the most important ones and slowest ones is due to the “normal” path option being selected. Try de-selecting it and see if it works.

This speeded it up a lot, and after I disabled my Real-Time AV protection (see below) the actual compression began after 22 seconds.

@spwolf:

This is the same problem as mentioned with that other thing in support email you sent, PowerArchiver has to workaround DynaZip in several cases for correct path option to be recorded properly.

Have you let DynaZip know about this problem ? It would be much easier (and be much faster in operation) to incorporate the “normal” option within DynaZip instead of having to “hack” it. When this option is set in PA, you’d simply set the option in DynaZip. I always prefer to have the “normal” option enabled, and so this massive slowdown is a BIG problem IMO. DynaZip should fix it!

AV scans every file during preparation stage, also causing massive slowdowns

It also appears that during the preparation process, PA (or DynaZip) opens the file (or does something) which causes my AV (KAV) to scan each file. Why does each file need to be opened ?

I don’t like having to disable my Real-Time AV protection so can this also be looked into please.

No files are scanned during WinZip’s preparation stage (so I assume it doesn’t open them).

deipotent

@deipotent:

AV scans every file during preparation stage, also causing massive slowdowns

It also appears that during the preparation process, PA (or DynaZip) opens the file (or does something) which causes my AV (KAV) to scan each file. Why does each file need to be opened ?

I don’t like having to disable my Real-Time AV protection so can this also be looked into please.

No files are scanned during WinZip’s preparation stage (so I assume it doesn’t open them).

This only occurs when “normal” option is disabled. When enabled, the files aren’t scanned by the AV.

spwolf

Hi,

You are correct - it is definetly an DynaZip issue and Yes, we have been trying to pressure them into giving us additional option to select paths in either way, nativly. Their point of view is that what PowerArchiver uses without “normal” option selected, is what it should be there… And we can “work around” it if we want, so they are pretty happy about providing both possibilites.

Of course, we shall try to continue to pressure them into giving us another option for path storing.

I have already noticed that in some cases (single file selection, full path, for example) we are using this even if we dont need to. So we will go and tripple check if everything is as fast as we could make it possibly right now, it gets confusing when you have 12 possible combinations to work with depending on user selection.

I will also check for AV scanning, I have to check deeper before giving you more info.

Few more questions - what is your current configuration? And are you using full path as well?

thanks

spwolf

Try another thing please:

-Move your folder to the 2 level deep subfolder
C:\test\realfoldertest\

Make sure full path option is delelected/try with selected as well
Right click on “realfoldertest” and compress it

Is that actually faster, even with “normal” path option selected?

deipotent

@spwolf:

You are correct - it is definetly an DynaZip issue and Yes, we have been trying to pressure them into giving us additional option to select paths in either way, nativly. Their point of view is that what PowerArchiver uses without “normal” option selected, is what it should be there… And we can “work around” it if we want, so they are pretty happy about providing both possibilites.

For the DynaZip people to say the “normal” option should not be there is a load of rubbish. Whatever happened to giving the user options, especially for a software component. When you next pester them about it, can you mention the statistics for when it is enabled, and also for when it is disabled. For them to say that you can workaround it, when clearly, it can’t be worked around (unless you want to wait an age when compressing large archives), is also a load of rubbish.

It wouldn’t even be that hard for them to implement.

You can also include a link to this thread.

@spwolf:

Few more questions - what is your current configuration? And are you using full path as well?

XP SP2 with Pentium III Mobile 1.2GHz and 512MB RAM.

I wasn’t using full path. If I choose full path, the compression starts after about 24-30 seconds, which is OK.

@spwolf:

I will also check for AV scanning, I have to check deeper before giving you more info.

When full path is enabled, and “normal” is either enabled or disabled, the Real-Time monitor also checks every file being added. Surely when full path is enabled, the “normal” option is ignored.

@spwolf:

Try another thing please:

-Move your folder to the 2 level deep subfolder
C:\test\realfoldertest\

Make sure full path option is delelected/try with selected as well

Right click on “realfoldertest” and compress it

Is that actually faster, even with “normal” path option selected?

Tried it already, and no it doesn’t speed it up.

spwolf

There was some work done to make this faster in 9.11.

In this specific case, complete fix/improvement will be made in 9.2, however even now you will be able to see improvements as well although it is still slow in this case no matter what. (IE in my example it went down from 26minutes to 7 minutes, and in 9.2 should be around 1-2 minutes which will be ok).

In some other cases, we have speeded it up a lot, so some people might see greater improvements.

p.s. deipotent - issue with reporting “in use” files from root folder is solved as well.

deipotent

Thanks for the work you’ve done on it. My test folder now takes 7 minutes before the compressions actually begins.

Please keep on pestering the DynaZip people to include support for this natively, since then you could have maximum speed during preparation.

In the meantime, I look forward to PA 9.2 for an even bigger improvement.

deipotent

The problem with Anti-Virus RealTime monitor scanning all files during preparation still exists in 9.11.01. I have “normal” option enabled and full path disabled, but it still occurs when these options are disabled and enabled respectively.

I’m using KAV Personal 5.0.227 with extended databases and Real-Time protection set to Maximum.

You can download a 30 day trial version of KAV from KAspersky’s website for testing purposes.

spwolf

@deipotent:

The problem with Anti-Virus RealTime monitor scanning all files during preparation still exists in 9.11.01. I have “normal” option enabled and full path disabled, but it still occurs when these options are disabled and enabled respectively.

I’m using KAV Personal 5.0.227 with extended databases and Real-Time protection set to Maximum.

You can download a 30 day trial version of KAV from KAspersky’s website for testing purposes.

We will see more specifics when 9.2 testing starts and since we will be changing things anyway in this situation (on how compression is done). So I will be able to tell you more what happens and why (most likely it wont anymore actually).

We will continue this discussion once testing starts since now I can only speculate things and not check them properly…

thanks,

deipotent

Okey dokey…

deipotent

A bit more info regarding the KAV problem:

I used Filemon (www.sysinternals.com) to see what the difference was between PA and WinZip. The result is that PA actually opens the file and closes it, which is obviously what’s causing KAV to scan the file. WinZip on the other hand only queries the files, but does not actually open it, hence KAV does not scan the file in this case.

I know PA 9.2 will be released for alpha testing soon, but thought I’d let you know about this extra info now.