[twilight-devel] Twilight filesystem layout idea

[Joseph Carter]

On Tue, Feb 25, 2003 at 05:24:32PM -0800, Forest 'LordHavoc' Hale wrote:
> Knghtbrd insists on a unified filesystem behavior across platforms, and 
> I can't see that as a reasonable possibility by any means, this is my 
> current proposal, which does not accomplish that feat but combines the 
> proposals I have seen so far...  It comes in two parts, the upper 
> majority of this email is the data path, then the profiles path, there 
> is also a note about menus at the end.

Well, the first goal is to fix the non-self-contained problem.  Twilight
is one of exactly TWO engines which is not self-contained on all
platforms.  While this is deemed the Linux Way(TM) and clasically the UNIX
Way(TM) as well, the reality is that UNIX vendors almost universally put
their entire program into a single directory, though the details of how
they make it work are vast, varied, and usually a hack.  Even so, the
reason it is done - to simplify things for the third party vendor - is a
valid consideration, especially since our users expect it already.  We
have, I think, reached consensus on this issue.  Noteworthy are Zeph's
reservations that Twilight must be able to work in a Linux Way(TM) setup
or certain distributions will get their panties in a bunch over it.


Without getting into technical details too much for a moment, I'm going to
follow the train of thought here first.  I managed to explain it to you
already, but I know for certain there are list subscribers who are unaware
of the reasons.  My thinking was going far, far beyond directory layout,
though it's the first step..  The ideas have refined a bit since then.

Problem:  We developers rarely install Twilight to the filesystem, we just
          run it from inside the source tree.
Solution: So make that the self-contained directory then..?

Problem:  We have a sub-directory named qw which contains code, not
          gamedata, causing namespace clash.
Solution: Put the gamedata in a sub-directory named data.  This would be
          even better for some, since it would allow data to be a symlink
	  to wherever you really keep gamedata instead of making copies,
	  for OSes which support that feature.  It'd also allow us to add
	  other sub-directories to the engine's gamedir later without
	  having the game think they are possibly playable mods..  The
	  advantages convinced me it was the right solution.

Problem:  Zeph points out that Windows users who unpack Twilight into
          their existing Quake directory will NOT be happy if Twilight
	  can't cope.
Solution: If data cannot be found in the gamedir, fall back to the gamedir
          itself.  This may disable some features which don't exist yet,
	  but we'll worry about that when and if we get there.

Problem:  And where was that gamedir again?
Solution: Well, in Win32 it is ., the current working directory!

Problem:  Uh, the current solution, as annoying to n00bs as it is, was
          devised specifically because . is NEVER the correct dir for
	  anything but Win32.
Solution: A function is needed which does its best to figure out where the
          program should look for things.  A good place to start for
	  non-Win32 machines is the location of the binary.  Unfortunately
	  this function will be #ifdef HELL because there is basically no
	  standard way to do this.  Many things should be tried, including
	  checking an environment variable and relying on defaults for the
	  OS in question set by ./configure (or whatever..)

Problem:  Okay, so you found the gamedir.  Now how exactly do you deal
          with multiple users, running the game from CD, etc?
Solution: Extend the path console command.  You should be able to add
          something to the beginning or end of the path, or simply just
	  change it.

Problem:  Change it!?  Tweaking the path could seriously screw with your
          ability to load files necessary for the engine to even RUN!
Solution: The engine doesn't need many files.  A console background, a
          font, a palette, that's it.  We know better fonts exist than the
	  Quake default, and the NQX console was basically one of the
	  nicest in any engine.

Problem:  You can't do much with an engine in that state.
Solution: Ahh, but you can!  You can rcon a server, you can configure the
          engine, including the location of your gamedata, or you can even
	  connect to a server if it will actually send you the files you
	  need to play..

Problem:  OpenQuartz is basically non-useful, and even Twilight's QW won't
          send you certain files needed to play.  NQ can't send you
	  anything at all.  The downloading all of the gamedata needed to
	  play is unlikely to ever happen.
Solution: Well, it could be done that QW could learn to send Twilight
          clients the shareware game if they need it.  In fact, it could
	  learn to send out a pak file (or preferably a zip file at some
	  point) instead of a bunch of little files.  NQ downloading was
	  never added because ... well, because NQ doesn't have a
	  connected but not active state.  If we wish to add one, we could
	  do that.  Regarding OpenQuartz, yeah it sucks, but at least it
	  needs to become possible..

Problem:  You want to be able to change the path AFTER the game starts?!
          That would totally break how Quake handles file management.
Solution: Yeah, but that needs to be fixed in order to be able to do
          things like change resolutions on the fly anyway.  The result
	  would be a resource manager capible of reloading every single
	  file used in the engine if necessary.  Of course, it would only
	  do so if necessary, namely if looking up a resource would have
	  changed it because the path was changed by the user or by an
	  EBSP or whatever..  There's no agreement that the EBSP should be
	  first, but this makes the most sense, and this sort of path
	  independence makes it possible.

Problem:  Stats for each file in each place it could possibly be, with an
	  arbitrary number of such places being added and removed all the
	  time will make things slow.
Solution: That's part of the resource manager's problem.  I have thoughts
          how to cache the location of the best location of a file, but
	  that will be regenerated each time the map loads.  Files needing
	  a reload will get it, but only those.  This will also result in
	  the ability to change mod at will, something that QW is hackish
	  about and NQ doesn't allow at all.

Problem:  Okay, so how do you deal with writing configs then?  You can't
          write to a PAK file (or at least you shouldn't) and Quake only
	  tries to write a file once.
Solution: Well make it smart enough to not do that, duh!  Most file ops
          just fail quietly in Quake and in Twilight too.  We should
	  replace the "assume it works and things just break subtly if it
	  doesn't" with an active error recovery condition.  In fact, path
	  elements that are compressed should be deemed read-only, as
	  should any directory in the path which is not writable.  The
	  path will be used for writing as well as for reading.

Problem:  That still doesn't address multiple users on MacOS 9 or Win9x.
Solution: That's true.  But since gamedata is assumed to live in its own
          sandbox, we can create a directory called profiles in the main
	  gamedir and allow you to make changes per "user" per mod that
	  way.  It also deals forever with the issue that we want to be
	  able to let people have configs which use twilight-specific
	  features, some of which already do exist.

Problem:  Where are you going to put logs, screenshots, etc?
Solution: In your profile, by default.  Of course, your profile can change
          these values.  This deals with another issue: the need to have
	  two gamedirs in order to run two servers using the same mod on
	  different ports.  Just give each server its own profile.  This
	  is a reason to have profiles whether you're on a multi-user
	  system or not.

Problem:  There isn't exactly an army of Quake players anymore, and most
	  of them don't really want all these changes anyway.  Some people
	  who make mods might think it's cool, but #qc will probably have
	  kittens when they see the engine-specific changes we've got in
	  mind, if we do them..
Solution: I don't know..  =(  I guess the real question is, would people
          actually be excited about making something that wasn't quite
	  Quake anymore?  You don't really mess with Twilight anymore,
	  neither does Zeph.  And we all know that I am hopeless when it
	  comes to time to do any of this, though I've got more interest
	  in doing something cool than having a rather vanilla Quake with
	  a bit of eye candy..


> P.S. I do see how it could be done almost identically across platforms 
> using profiles, see notes at end.

I'm refining the thinking a bit to conclude that the end result (interim
attempts may do various evil things I fear), will simply do what it needs
to do in order to get you a sane default path.  I'm not sure what that
entails yet.

Of course, I didn't want to get into the gory bits up there, but the idea
seems to be to create a master path of gamedirs, which is what you edit.
The rest is just handled for you.  This changes the path command in an
unexpected manner, but I'm not sure it won't be found to be a good change
in the end.  Still working out some of those kinds of details.


> To be clear, the data path is where the id1, ctf, fortress, etc 
> directories reside, the profiles directory has one directory per user 
> profile (on UNIX each user might have multiple profiles), and each of 
> those directories contains mod directories (containing configs, 
> screenshots, logs, demos, etc).

This seems logical and has since the moment you mentioned it.


> Note that I still do not like the idea of detecting the executable path, 
> but I'll run with the idea anyway, since knghtbrd will insist on it.

I'm thinking that it might not be necessary to do it except in the absense
of something else which tells us what we need to know, which is similar to
your plan to simply get the information and then store it, except that we
don't need to ask if the user didn't do something dumb.

I wish I'd thought through the path thing before since it will really make
this much easier in the end, and much more sane.  Not ready for a proposal
of a final result, and even if I were, that's probably not what I'd wind
up coding in the interim, since there's too much interdependent code in
the mix.

At some point it would be nice to have a cross-platform file dialog...
hmm.

> If that failed...
> A: (UNIX)
> puts up an SDL based messagebox asking where the data is and adjusts the 
> ~/.twilight/data link to match, then starts the game.

I'd rather just start the console with an error message telling them what
to do.  Of course, that means the engine would need to become a bit more
DWIM about handling the no data case, but we want that anyway.  No user
should ever again have to ask me where his gfx.wad is..  =p


> B: (UNIX)
> console prompt asking where the data is and adjusts the ~/.twilight/data 
> link to match, then starts the game.

That won't work with downloads, really.


> C:
> same as fallback A except uses EXECUTABLEPATH/datadir.txt file or 
> something along those lines instead of ~/.twilight/data. (yes this may 
> fail to write the file - it will try to write an empty file there before 
> putting up prompt to be sure this works)

A symlink for OSes that don't have one..  Oh that is just sick.  ;)  It
might get used, too!  This could be the answer to the question of how to
deal with users not wanting to have a data subdir because it'd break their
existing quake setup...


> D:
> puts up an SDL or native messagebox, which reports that no data was 
> found, and all the places it tried.

I think the message should include either the attempted locations as you
suggest, or enough information so you can find them.


> Possible configurations for finding a profiles directory (tried in 
> sequence at startup):
> A:
> fs_path_profiles "profiles/"
> B: (UNIX)
> fs_path_profiles "~/.twilight/profiles/"
> C:
> fs_path_profiles "EXECUTABLEPATH/profiles/"
> 
> If that failed...
> Possible configurations for creating a profiles directory (tried in 
> sequence):
> A: (UNIX)
> fs_path_profiles "~/.twilight/profiles/"
> B:
> fs_path_profiles "EXECUTABLEPATH/profiles/"
> C:
> fs_path_profiles "profiles/"
> D:
> error out with a messagebox and/or console message about being unable to 
> find or create a profiles directory

I really would like to get away from the won't start case.  I should say
something here about the master path, but I'm not really in the position
to do that yet since I haven't thought about it sufficiently.


> Now as for the nature of ~/.twilight/data (which of course is only on 
> UNIX), if a user installed by using the installer, this is a data 
> directory or a link to data, it may additionally be altered by the data 
> path failure mechanisms (if it asks for the location).
> 
> I would also like to strongly suggest that we need to make twilight 
> completely capable of running without a data directory of any kind and 
> able to switch data directories at any time.  This would give us great 
> flexibility in handling of failure conditions and be a good idea 
> all-round.  (As a bonus this means NQ could switch mods, we'd want to 
> add a mod browser to the menu)

We've talked about this before, and we've even considered the result of
doing it once or twice.  I agree though, it's the solution of choice.


> This does of course depend on embedding dpmviewer/lhfont.tga into the 
> twilight executables (it's only 7311 bytes as BGRA RLE targa, could be 
> made much smaller if needed), and redoing the menu system (come on, it 
> needs a rewrite, even if we end up completely changing the look in the 
> process).

I haven't seen this targa, but I suspect the internal font would probably
be another 8x8 font, probably the pixel font I did which came out so
crisp..  Mostly because I meant what I said about the engine coming up
no matter what - even if that means in software.  Just don't think you are
gonna connect to a server as a player until you restart video in OpenGL.
;)


> P.S. the existence of ~/.twilight worries me, because it is an 
> unnecessary association with the engine name - you might have multiple 
> games installed, each with their own data paths, all using twilight. 
> One possible solution would be storing a datadir.txt or whatever per 
> profile, and allow it to be edited by the game, this allows multiple 
> entire games to be lumped together in the profiles directory, start 
> twilight and choose a profile, which also chooses a game.  Comments?
> 
> P.S. This could eliminate the whole data search path for most purposes, 
> we'd only need to locate the profiles directory (easy!), and let the 
> user configure data paths ingame.

The goal was to locate twilight.cfg if it exists and use that to get
everything else set up.  If not, start looking for things by checking for
it in the most obvious places for the OS in question and using defaults
when you can't find what you want.

-- 
Joseph Carter <knghtbrd at efn.org>             A mighty Oak is the result of
  ^ a nut                                       a nut that held its ground
 
!netgod:*! time flies when youre using linux
!doogie:*! yeah, infinite loops in 5 seconds.
!Teknix:*! has anyone re-tested that with 2.2.x ?
!netgod:*! yeah, 4 seconds now

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 253 bytes
Desc: not available
URL: <http://icculus.org/pipermail/twilight-devel/attachments/20030226/c49898dd/attachment.pgp>