[physfs] mmap'ed files

fungos fungos at gmail.com
Tue Feb 7 15:36:05 EST 2012


kOn Tue, Feb 7, 2012 at 3:23 PM, Ryan C. Gordon <icculus at icculus.org> wrote:
> On 02/06/2012 03:05 PM, fungos wrote:
>>
>> I think that mmap'ed files aren't implemented on 2.0.2 right?
>
>
> No.
>
> We can't offer an mmap()-like API for accessing files, because it wouldn't
> be helpful for files inside archives that might be stored as compressed
> data.

But it could be an option for uncompressed packages.

> While we _could_ mmap() an entire archive behind the scenes, it's not really
> safe to do so unconditionally, because a multi-gigabyte .zip file would eat
> all of the address space of a 32-bit process.

I don't think this should be something automagically, but user guided.
See the iOS case, Carmack says:

"Interestingly, you can only memory map about 700 megs of virtual
address space, which is a bit surprising for a 32 bit OS. I expected
at least twice that, if not close to 3 gigs. We sometimes have a
decent fraction of this mapped."

http://www.bethblog.com/2010/10/29/john-carmack-discusses-rage-on-iphoneipadipod-touch/

Well, this is true, we can't assume that any package can be mapped,
but the user could hint which ones can or should.

> The best bet here might be to mmap() smaller files behind the scenes (or
> mmap() everything in a 64-bit process). Alternately, the 2.1 branch lets you
> supply your own i/o implementations, so you could use PHYSFS_mountIo() to
> load an archive and work out of an mmap()'d buffer.

I don't like to mmap smaller files, they aren't the real problem and
will not pay off.
About the mountIo API, this is a good option, but not so versatile.
I think it is easier to use case-by-case, like mounting one package
explictly specifing that it should use mmap.

> All that being said: we don't really load a whole archive to RAM...mostly we
> read small pieces from it as necessary, so it's not necessarily much of a
> drain on system resources to _not_ mmap it.

Don't know if this hold true for huge games.

I want a texture pack (unpacked package) with tons of pvrtc textures
inside, something like 500mb of textures, and I will have a lot of
textures in use in the same time. I think it is not possible to
accomplish this without mmaped files, is it?
My problem is that I don't want to load textures to heap so then I can
upload it to the opengl, this is really not optimal, this impose a
limit of 4 png-texture-atlases of 1024x1024 at best without ios
killing our app or sending memory warnings.

Thank you


More information about the physfs mailing list