[physfs] Failures loading from some (but not all) zip files with particular filenames

[Andi McClure]

I have been using physfs to load data out of .zip files. I
periodically see a failure where there is a file which is definitely
in the .zip file, but which physfs cannot open. In every one of these
cases, the file turned out to be accompanied in the same zip by a file
of similar name, but with the additional suffix ".orig" (these files
are created by mercurial and sometimes I accidentally include them in
my builds); removing the .orig file has always made the problem
magically disappear. I have seen this definitely three times, maybe
more, two of these times this weekend. Note this does not happen
*every* time a zip contains a .orig file, just, occasionally I
randomly wind up with a zip file with a .orig file and also this
problem. I have managed to actually catch one of the .zip files which
provoke the failure and made a little demonstration program showing
the problem.

Attached to this email are two files, "broken.zip" and "working.zip",
containing little source directories. The contents of the two
directories are identical except for a file physics3d.pak (this is a
.zip file that has been renamed to ".pak"). If you run make in either
source repo (you may need to edit the makefile to make it compile on
your computer; I run "make MCC=1" to tell it where to find physfs and
Foundation and such), it will unzip physics3d.pak, and build four
programs:

dirtest_open: Attempts to load a single file PhysicsScene.lua from the
unzipped directory and print its first 100 bytes.
dirtest_scan: Using PHYSFS_enumerateFiles, attempts to load every file
from the unzipped directory and print their first 100 bytes.
ziptest_open: Attempts to load a single file PhysicsScene from the zip
and print its first 100 bytes.
ziptest_scan: Using PHYSFS_enumerateFiles, attempts to load every file
from the zip and print their first 100 bytes.

dirtest_open and dirtest_scan will always succeed. If you run
ziptest_open or ziptest_scan out of the "working" repo, they will also
succeed; if you run them out of "broken" you will get a "Couldn't
open" error when it attempts to call PHYSFS_openRead on
PhysicsScene.lua (but all other files will succeed). (Incidentally, if
I add a PHYSFS_getLastError right after the failed openRead, the error
printed is "File not found".

Several things strike me as unusual here.

1. The only difference between "broken" and "working", again, are the
file physfs3d.pak. However the two versions of physfs3d.pak contain
the exact same files. The physfs3d.pak in "broken" was created by a
shell script which runs as part of the build process for one of my
projects, which just called the linux "zip" tool to create it
(although I have also seen this problem running with OS X); it's done
this dozens of times without incident, but this one time it emitted
this zip physfs doesn't like. The physfs3d.pak in "working",
meanwhile, was created by (a) unzipping the bad physfs3d.pak (b)
mv-ing PhysicsScene.lua out of the directory (c) mv-ing it back in (d)
deleting and re-zipping physfs3d.pak. Notice, *steps b and c were for
some reason necessary*. If I simply unzip, then rezip, the
physfs3d.pak from the "broken" repo, the error does not go away. This
tells me the problem is not simply a corrupt zip file.

2. in the "broken" version of the ziptest_scan test, although physfs
insists the PhysicsScene.lua test is not found when I open it, it
still lists the file in PHYSFS_enumerateFiles. It seems as if it
should be considered a bug if it is ever possible for enumerateFiles
to list a file for which openRead would return "not found".

Do you think, can whatever in physfs is causing these failures to
occur be fixed? Every time I run into this problem it is a potentially
large block of lost time because the errors it produces are difficult
to identify (i.e. files fail to open which I should be able to
absolutely assume are present and therefore didn't write in error
handling for; also because in my personal build process I don't use
zips when debugging, for me the failures show up only in release
builds, which are harder to debug).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: broken.zip
Type: application/zip
Size: 7292 bytes
Desc: not available
URL: <http://icculus.org/pipermail/physfs/attachments/20120409/73ab6567/attachment.zip>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: working.zip
Type: application/zip
Size: 7101 bytes
Desc: not available
URL: <http://icculus.org/pipermail/physfs/attachments/20120409/73ab6567/attachment-0001.zip>