[physfs] 7z/lzma in trunk

Dennis Schridde devurandom at gmx.net
Tue Sep 26 14:30:04 EDT 2006


Am Freitag, 1. September 2006 21:45 schrieb Thomas Einsporn:
> > So it will take some time till I got an answer and longer till that will
> > be working in physfs... Please be patient. ;)
It took me quite a while till I figured out how the lzma sdk is supposed to 
work but after that I got a simple solution.
I hope it is bugfree and at least found no crashes when trying with some 
sample archives.
All of them were compressed with standard (read: no extra) options.
The test was compiled on Linux using GCC, so it would be nice if this got 
tested on Windows with MSVC.


This is not a small bugfix but rather a whole reworked version using many more 
of the LZMA SDKs functionality.


A patch is attached. Please note that I removed some files and added others.
D      lzma/LzmaStateDecode.c
D      lzma/LzmaStateDecode.h
A      lzma/7zExtract.c
A      lzma/7zExtract.h
A      lzma/LzmaDecode.c
A      lzma/LzmaDecode.h
A      lzma/LzmaTypes.h


The SDK has also been updated to 4.43 and is (nearly) a non modified version:
In files 7zIn.c:185 and 7zDecode.c:73 I had to change inBuffer to void* 
because otherwise GCC4 complained about dereferencing type-punned pointers.
See also the bugreport: 
http://sourceforge.net/tracker/index.php?func=detail&aid=1565800&group_id=14481&atid=114481


In case others want to know how this works:
A 7z "folder" means a block of the compressed archive.
SzExtract extracts one complete block and gives the offset of the requested 
file in that block.


My implementation caches one block per archive. You can experiment with the 
blocksize of your archives to find out what gives you the best speed / memory 
usage ratio.


It could be changed to cache more than one block and switch between them on 
read() in case where you need random access on big archives.
In case where you only read one file at a time the current code will be 
sufficient (IMO).
In the multi cache approach LZMA_read() would keep the last X cached blocks.
So when it finds that the folder (read: block) of the requested file is 
currently cached, it would increase the priority of that block.
If the folder of the requested file is currently not in cache it would drop 
the cached block with the lowest priority (read: least frequently used) block 
and instead cache another block.
But as this probably needs quite some fine tuning (number of cached blocks and 
priority algorithm) I didn't implement it. (And as I said, the current 
approach is probably sufficient.)


I hope this makes it's way into SVN soon. :)

--Dennis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: reworked_lzma.patch
Type: text/x-diff
Size: 86317 bytes
Desc: not available
URL: <http://icculus.org/pipermail/physfs/attachments/20060926/76d6b63e/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://icculus.org/pipermail/physfs/attachments/20060926/76d6b63e/attachment.pgp>


More information about the physfs mailing list