Smart handling means what you wrote it means, however with certain files, it can not detect smartly because they actually allow reading sometimes and up to certain level.
this is why you can use the filter to force it.
VSS is used for enterprise level backups usually, and they simply always use it. However since PA is different kind of utility, we cant do that as first init of Volume Shadow Copy Service can take anywhere from 8s to 1 minute, depending on the computer.