Announcing MojoSetup.

[Ryan C. Gordon]

...so I set out to revamp some pieces of loki_setup to make progress on 
a 2.0 release.

We first talked about 2.0 in June of 2005:

http://icculus.org/cgi-bin/ezmlm/ezmlm-cgi?2:mss:666:200506:fjepgpbjejjcppnfihnf

And the latest commit to the 2.0 branch happened around that same time:

http://cvs.icculus.org/cvs/loki_setup/?sortby=date&only_with_tag=SETUP_2_0

Several of us have been submitting patches to 1.0 in the meantime, 
making good fixes and adding workarounds for deficiencies but not really 
making improvements to the codebase.

Generally loki_setup has worked well enough so it wasn't worth investing 
the effort to do radical improvements, but there are many many areas 
where it is either annoying, outdated, or flat out broken. Some of these 
were issues of lack of engineering resources (we're all busy as hell, 
and usually desperately patching the installer when issues crop up 
during crunch time on a product shipping in the next several hours!), 
and some were things that broke as Linux progressed (glibc 
incompatibilities, etc). Some were legitimate design decisions that 
turned out to be less than ideal in the real world. The codebase is very 
organic, and you can see where various vines of code grew based on 
concessions made over time. Evolution is a painful process.   :)

So I started out to do heavy revamping of loki_setup to get to 2.0, 
starting with the archive plugin stuff:

http://icculus.org/cgi-bin/ezmlm/ezmlm-cgi?2:mss:678:200506:nbooliokgneeciafbijn

...and as I started to think about the implications of changing that 
subsystem, I realized that there were other related things I'd like to 
rework, too, and eventually I realized I wanted to rework almost 
everything. By the time I mapped out an ideal loki_setup, it wasn't 
loki_setup at all anymore.

So I started from scratch.

I've been making good progress on this (I'm calling it "MojoSetup" at 
the moment), and while there is still much to be done, I feel it's 
finally ready to show to the public. Here are some things that panned 
out pretty well so far...this is a _lot_ of information, but I thought 
you might be interested in the details, and maybe have some opinions to 
share.

Most of MojoSetup was designed explicitly to solve specific shortcomings 
in loki_setup. It's not that I'm trashing on loki_setup, it's just what 
I think a nextgen replacement would have to address.

Some of these are technical notes, some are bullet point features. Skip 
the ones that bore you.   :)   All of these are currently implemented, 
but some still need work (the UI could use some polishing, for instance).

- MojoSetup uses Lua, a lightweight scripting language embedded in the 
installer, for most of the work: the localization tables are a Lua 
script, the config file is a Lua script, and a good portion of the 
actual installer code is Lua script...several pieces of data that the C 
portions need are stored in Lua tables so they benefit from its garbage 
collector. I was looking at libxml, which loki_setup uses, and found xml 
pretty limiting...you have to over-define everything, and do some really 
awkward things when a quick "if" statement in a script would be way 
easier for everyone...reference this in the example config file in 
loki_setup:

              <option if="+(+(x86,Linux),has-passwd,!false)">

I don't even know what that does!

There are lots of other attributes in setup.xml that were clearly 
special cases that would be better in installer-specific logic, rather 
than forever locked into the config file schema and bloating the 
installer itself. Not to mention some attributes needing to be "true", 
some needing to be "yes", some just needing to exist...the schema and 
the interpretation of it had its share of problems. A scripting 
language, even one as liberal as Lua, tends to enforce correct syntax 
much better.

Having a scripting language instead of an XML schema lets us keep 
special cases out of the installer. If you have some strange case (like 
what the "libc" tag does in loki_setup), you can roll your own test in 
the config script without bloating the installer:

           if (MojoSetup.libc == "glibc-2.0") then
               install_glibc20_bins()   -- or whatever.
           end

Also, libxml2 turned out to be _insane_ for file size if you have to 
statically link it: for example, this test program...

            http://xmlsoft.org/examples/reader2.c

...which parses and validates an XML file and nothing else, is 992 
kilobytes when statically linked to libxml2 (on PowerPC Mac OS X, 
compiler optimized for size, no debug symbols). More if you statically 
link zlib and libiconv, both of which libxml2 requires. Lua as a whole 
produces a 144k binary that has a full parser, interpreter, and runtime 
library. If you chop out the parser, it's 120k, and if you strip out the 
non-essential runtime library bits, it's 72k (and smaller on x86)...and 
links against the C runtime and nothing else. All those chatty config 
files can optionally be compiled down to Lua bytecode before shipping, 
too, to reduce their size.

Not to mention the general installer logic doesn't need to be fast, but 
it DOES need some protection from the usual C problems of buffer 
overruns, uninitialized variables and memory management, so anything 
that can be reasonably pushed into Lua has been. Calling between C and 
Lua is, unlike every other scripting language I tried, very very easy, 
so you can jump back and forth between them as it makes sense to do so.

Overall, using Lua proved to be a big design win.


- The installer generally looks for data it needs and wants to install 
in specific places in a directory tree, but access to that tree is 
abstracted out, so it can be a real directory or a .zip archive or 
whatever. This leads to a few interesting scenarios:

1) You can have a standard "package format" (filename.mojosetup or 
whatever), that you could distribute if you expect MojoSetup to be 
preinstalled on the system. That gets around the 
executable-bit-on-downloads issue if distributions show up and install 
MojoSetup...but this isn't required.

2) You can put your data in a zip file, and append it to the installer 
executable. Since the installer doesn't need to unpack itself first to 
access files like makeself/loki_setup does, you can run the installer 
and it can open _itself_ like a zipfile, and read from a file hierarchy 
without writing anything to disk first. This is a win for two reasons: 
usually loki_setup packages have to be downloaded (one copy to disk), 
unpacked (two copies, maybe an uncompression stage), run, and THEN the 
real installation starts (three copies, maybe a second uncompression 
stage). MojoSetup can avoid all that...zipfiles allow random access, 
unlike tar files, so it can pretty much start up and respond to the user 
immediately, and mostly only write files when really installing. At the 
simplest level, building an installer is just a matter of using a 
standard .zip utility. You can use other formats besides .zip, this is 
just the obvious choice in this case.

3) Support from installing from archives inside archives. The basic 
design, by necessity, had to treat a zipfile appended to a binary like a 
real filesystem that might contain tarballs that need unpacking, etc. 
Once that abstraction was in place, it cleanly solves a problem I ran 
into on "Cars: Radiator Springs Adventures" ... it had to use the same 
install data as the Windows InstallShield installer on the disc...for 
whatever reason, InstallShield packaged a .zip file of the game data 
inside a .jar file! The loki_setup installer had a seriously mangled and 
customized zip.c for this. In MojoSetup, your config would just need to 
say something like...

     setup.File {
         source = "media://retail-disc/install.jar/gamedata.zip";
     }

...and the installer does the Right Thing.


- Part of the workaround for incompatible and missing GUI libraries in
loki_setup was to statically link everything you could, and ship
multiple copies of the installer...a GTK2 version, a statically-linked
GTK1 version, an ncurses version, etc...there's a whole shell script in
there to try them all until one actually starts up, since the assumption
is that the GTK+2 one may be missing a Gnome dependency, etc. It doesn't
help that each of these version's codebase is mostly a cut-and-paste of
some other version--including generic installer logic--and bugfixes and
improvements applied to one don't help the others. It made the download
big and startup flakey.

MojoSetup uses plugins for the UI. It has a very tightly-defined 
interface for what the plugin must do, and keeps all generic logic out 
of them. While a plugin can be statically linked to the installer (which 
generally is done for, say, the stdio plugin), most are in their own 
shared library, which MojoSetup tries to load at startup from inside the 
installer package. The plugin may fail to load (gtk plugin and there's 
no GTK+2 libs on the system, etc), and MojoSetup will just go on to the 
next. Since we don't have to worry about the whole application failing 
to startup, we never have to statically link a whole GUI toolkit, nor 
spend effort on trying to dlsym() all the functions we need in GTK+, 
just in case. Once a plugin loads, it can tell MojoSetup what priority 
it should be ("I can't connect to an X server, never use me," "I'm a Qt 
plugin and I can talk to an X server, so I'll work, but you aren't 
running KDE, so try me last," "I only write to stdout, try me absolutely 
last," etc). Users can override the choice of GUI plugins if they like 
from command line or environment variable. This keeps the download 
small, lets us provide a bunch of options without worry, and keep the 
codebase clean.

When I shipped a Linux version of Candy Cruncher (a casual game from 
pyrogon.com), I had loki_setup with a statically linked GTK+1 UI, and a 
statically linked curses/dialog UI for the installer, wrapped in a 
makeself script. I got complaints from dial-up users that the 
installer's download was bigger than the installed game. In MojoSetup, 
the GTK+2 UI currently adds a whopping 14 kilobytes to the download 
_before compression_. The stdio UI is 6. We could probably drop in a 
KDE, SDL, and wxWidgets implementation for about the same space and let 
the installer sort out the best plan of attack on the end user's system. 
Compressed, all these options combined would probably add about 30 to 40 
kilobytes to the download.


- All strings are UTF-8 internally. Unicode and localization were 
considered from the start. All translations are kept in a single text 
file (a Lua script, actually), so you don't have to fight with GNU .po 
files.


- The UI looks more like Apple's installer (well, it will!). It asks a 
few basic questions and then does its thing with a "wizard" style 
interface. It looks more "modern" than loki_setup (well, it will!).


- There is a "rollback" mechanism. Barring catastrophic disk failure, 
failed installations can undo everything they wrote to the filesystem, 
including restoring preexisting files that were overwritten during the 
install. The installer even makes an earnest attempt at cleanup and 
rollback if it is crashing with a segfault.


- As you would expect, the installer can use CD and DVD discs (and USB 
keychain drives, Samba shares, etc) for installation media, but it can 
also use data stored on a web server. You can specify files to be 
obtained at runtime over HTTP or FTP, which MojoSetup will do before 
starting the actual install loop. This is useful for shipping an 
extremely small package that gets the user to a UI as fast as possible, 
then downloads just the optional bits they choose to install...this also 
allows a vendor to supply updated installer packages on their website 
without having to repackage the installer itself.


- MojoSetup is under the zlib license. We had to put a hack in 
loki_setup for UT2003 to spawn an external application, since we 
couldn't link the UnrealEngine to loki_setup to verify CD keys without 
violating the GPL.  This is also useful for installing arbitrary data 
formats (like the Outrage package format on the Descent 3 expansion 
disc) for which the publisher wishes to keep proprietary...they can 
write a one-shot archiver without violating the GPL.

MojoSetup is useful for me, and I hope it will be useful to you, too, 
under whatever circumstances you use it.


- MojoSetup already targets Unix systems, with the initial grumblings of 
support for Mac OS X, Windows and BeOS (!). Other Unixes are trivial to 
target. Like the GUI plugins, the platform-specific bits are being 
carefully separated out. loki_setup really sort of assumes a Unix (and 
in some cases, a Linux) system. Config files are somewhat easier to 
reuse between platforms than loki_setup's...even if the most desparate 
scenarios, you could just have an if/else block for each platform, since 
the config file is a Lua script.


- No autoconf/automake nastiness, we're using CMake (thanks to KDE for 
moving to that and making it a viable tool).


- Development is done in a Subversion repository instead of CVS, in the 
same way that CMake was a newer, but better alternative to autotools.


- ...probably other things.  :)


You can grab MojoSetup from Subversion here:

      svn://svn.icculus.org/mojosetup/


...and get on the MojoSetup mailing list by sending a blank email to 
mojosetup-subscribe at icculus.org ... I figure MojoSetup will be of 
serious interest to loki_setup users, which is why I'm posting this 
announcement here, but ongoing discussion probably shouldn't happen on 
this list.

As a test project, I have a build of Duke Nukem 3D for Linux up for 
download:

      http://icculus.org/mojosetup/examples/duke3d/

The installer is a 132 kilobyte download, which can then either install 
the shareware version by downloading the data at runtime, or install the 
full game off a retail disc. The config file for the installer is in 
there, too, so you can get an ideal of what rolling your own would be like.

The UI is really rough at the moment and there are some other bugs, too. 
There are some obvious features of loki_setup that aren't yet available 
in MojoSetup, and the documentation is completely non-existent at this 
time...but it gives you an idea of what I'm going for here.

--ryan.