Ryan C. Gordon
icculus at icculus.org
Sat May 12 11:44:59 EDT 2007
...so I set out to revamp some pieces of loki_setup to make progress on
a 2.0 release.
We first talked about 2.0 in June of 2005:
And the latest commit to the 2.0 branch happened around that same time:
Several of us have been submitting patches to 1.0 in the meantime,
making good fixes and adding workarounds for deficiencies but not really
making improvements to the codebase.
Generally loki_setup has worked well enough so it wasn't worth investing
the effort to do radical improvements, but there are many many areas
where it is either annoying, outdated, or flat out broken. Some of these
were issues of lack of engineering resources (we're all busy as hell,
and usually desperately patching the installer when issues crop up
during crunch time on a product shipping in the next several hours!),
and some were things that broke as Linux progressed (glibc
incompatibilities, etc). Some were legitimate design decisions that
turned out to be less than ideal in the real world. The codebase is very
organic, and you can see where various vines of code grew based on
concessions made over time. Evolution is a painful process. :)
So I started out to do heavy revamping of loki_setup to get to 2.0,
starting with the archive plugin stuff:
...and as I started to think about the implications of changing that
subsystem, I realized that there were other related things I'd like to
rework, too, and eventually I realized I wanted to rework almost
everything. By the time I mapped out an ideal loki_setup, it wasn't
loki_setup at all anymore.
So I started from scratch.
I've been making good progress on this (I'm calling it "MojoSetup" at
the moment), and while there is still much to be done, I feel it's
finally ready to show to the public. Here are some things that panned
out pretty well so far...this is a _lot_ of information, but I thought
you might be interested in the details, and maybe have some opinions to
Most of MojoSetup was designed explicitly to solve specific shortcomings
in loki_setup. It's not that I'm trashing on loki_setup, it's just what
I think a nextgen replacement would have to address.
Some of these are technical notes, some are bullet point features. Skip
the ones that bore you. :) All of these are currently implemented,
but some still need work (the UI could use some polishing, for instance).
- MojoSetup uses Lua, a lightweight scripting language embedded in the
installer, for most of the work: the localization tables are a Lua
script, the config file is a Lua script, and a good portion of the
actual installer code is Lua script...several pieces of data that the C
portions need are stored in Lua tables so they benefit from its garbage
collector. I was looking at libxml, which loki_setup uses, and found xml
pretty limiting...you have to over-define everything, and do some really
awkward things when a quick "if" statement in a script would be way
easier for everyone...reference this in the example config file in
I don't even know what that does!
There are lots of other attributes in setup.xml that were clearly
special cases that would be better in installer-specific logic, rather
than forever locked into the config file schema and bloating the
installer itself. Not to mention some attributes needing to be "true",
some needing to be "yes", some just needing to exist...the schema and
the interpretation of it had its share of problems. A scripting
language, even one as liberal as Lua, tends to enforce correct syntax
Having a scripting language instead of an XML schema lets us keep
special cases out of the installer. If you have some strange case (like
what the "libc" tag does in loki_setup), you can roll your own test in
the config script without bloating the installer:
if (MojoSetup.libc == "glibc-2.0") then
install_glibc20_bins() -- or whatever.
Also, libxml2 turned out to be _insane_ for file size if you have to
statically link it: for example, this test program...
...which parses and validates an XML file and nothing else, is 992
kilobytes when statically linked to libxml2 (on PowerPC Mac OS X,
compiler optimized for size, no debug symbols). More if you statically
link zlib and libiconv, both of which libxml2 requires. Lua as a whole
produces a 144k binary that has a full parser, interpreter, and runtime
library. If you chop out the parser, it's 120k, and if you strip out the
non-essential runtime library bits, it's 72k (and smaller on x86)...and
links against the C runtime and nothing else. All those chatty config
files can optionally be compiled down to Lua bytecode before shipping,
too, to reduce their size.
Not to mention the general installer logic doesn't need to be fast, but
it DOES need some protection from the usual C problems of buffer
overruns, uninitialized variables and memory management, so anything
that can be reasonably pushed into Lua has been. Calling between C and
Lua is, unlike every other scripting language I tried, very very easy,
so you can jump back and forth between them as it makes sense to do so.
Overall, using Lua proved to be a big design win.
- The installer generally looks for data it needs and wants to install
in specific places in a directory tree, but access to that tree is
abstracted out, so it can be a real directory or a .zip archive or
whatever. This leads to a few interesting scenarios:
1) You can have a standard "package format" (filename.mojosetup or
whatever), that you could distribute if you expect MojoSetup to be
preinstalled on the system. That gets around the
executable-bit-on-downloads issue if distributions show up and install
MojoSetup...but this isn't required.
2) You can put your data in a zip file, and append it to the installer
executable. Since the installer doesn't need to unpack itself first to
access files like makeself/loki_setup does, you can run the installer
and it can open _itself_ like a zipfile, and read from a file hierarchy
without writing anything to disk first. This is a win for two reasons:
usually loki_setup packages have to be downloaded (one copy to disk),
unpacked (two copies, maybe an uncompression stage), run, and THEN the
real installation starts (three copies, maybe a second uncompression
stage). MojoSetup can avoid all that...zipfiles allow random access,
unlike tar files, so it can pretty much start up and respond to the user
immediately, and mostly only write files when really installing. At the
simplest level, building an installer is just a matter of using a
standard .zip utility. You can use other formats besides .zip, this is
just the obvious choice in this case.
3) Support from installing from archives inside archives. The basic
design, by necessity, had to treat a zipfile appended to a binary like a
real filesystem that might contain tarballs that need unpacking, etc.
Once that abstraction was in place, it cleanly solves a problem I ran
into on "Cars: Radiator Springs Adventures" ... it had to use the same
install data as the Windows InstallShield installer on the disc...for
whatever reason, InstallShield packaged a .zip file of the game data
inside a .jar file! The loki_setup installer had a seriously mangled and
customized zip.c for this. In MojoSetup, your config would just need to
say something like...
source = "media://retail-disc/install.jar/gamedata.zip";
...and the installer does the Right Thing.
- Part of the workaround for incompatible and missing GUI libraries in
loki_setup was to statically link everything you could, and ship
multiple copies of the installer...a GTK2 version, a statically-linked
GTK1 version, an ncurses version, etc...there's a whole shell script in
there to try them all until one actually starts up, since the assumption
is that the GTK+2 one may be missing a Gnome dependency, etc. It doesn't
help that each of these version's codebase is mostly a cut-and-paste of
some other version--including generic installer logic--and bugfixes and
improvements applied to one don't help the others. It made the download
big and startup flakey.
MojoSetup uses plugins for the UI. It has a very tightly-defined
interface for what the plugin must do, and keeps all generic logic out
of them. While a plugin can be statically linked to the installer (which
generally is done for, say, the stdio plugin), most are in their own
shared library, which MojoSetup tries to load at startup from inside the
installer package. The plugin may fail to load (gtk plugin and there's
no GTK+2 libs on the system, etc), and MojoSetup will just go on to the
next. Since we don't have to worry about the whole application failing
to startup, we never have to statically link a whole GUI toolkit, nor
spend effort on trying to dlsym() all the functions we need in GTK+,
just in case. Once a plugin loads, it can tell MojoSetup what priority
it should be ("I can't connect to an X server, never use me," "I'm a Qt
plugin and I can talk to an X server, so I'll work, but you aren't
running KDE, so try me last," "I only write to stdout, try me absolutely
last," etc). Users can override the choice of GUI plugins if they like
from command line or environment variable. This keeps the download
small, lets us provide a bunch of options without worry, and keep the
When I shipped a Linux version of Candy Cruncher (a casual game from
pyrogon.com), I had loki_setup with a statically linked GTK+1 UI, and a
statically linked curses/dialog UI for the installer, wrapped in a
makeself script. I got complaints from dial-up users that the
installer's download was bigger than the installed game. In MojoSetup,
the GTK+2 UI currently adds a whopping 14 kilobytes to the download
_before compression_. The stdio UI is 6. We could probably drop in a
KDE, SDL, and wxWidgets implementation for about the same space and let
the installer sort out the best plan of attack on the end user's system.
Compressed, all these options combined would probably add about 30 to 40
kilobytes to the download.
- All strings are UTF-8 internally. Unicode and localization were
considered from the start. All translations are kept in a single text
file (a Lua script, actually), so you don't have to fight with GNU .po
- The UI looks more like Apple's installer (well, it will!). It asks a
few basic questions and then does its thing with a "wizard" style
interface. It looks more "modern" than loki_setup (well, it will!).
- There is a "rollback" mechanism. Barring catastrophic disk failure,
failed installations can undo everything they wrote to the filesystem,
including restoring preexisting files that were overwritten during the
install. The installer even makes an earnest attempt at cleanup and
rollback if it is crashing with a segfault.
- As you would expect, the installer can use CD and DVD discs (and USB
keychain drives, Samba shares, etc) for installation media, but it can
also use data stored on a web server. You can specify files to be
obtained at runtime over HTTP or FTP, which MojoSetup will do before
starting the actual install loop. This is useful for shipping an
extremely small package that gets the user to a UI as fast as possible,
then downloads just the optional bits they choose to install...this also
allows a vendor to supply updated installer packages on their website
without having to repackage the installer itself.
- MojoSetup is under the zlib license. We had to put a hack in
loki_setup for UT2003 to spawn an external application, since we
couldn't link the UnrealEngine to loki_setup to verify CD keys without
violating the GPL. This is also useful for installing arbitrary data
formats (like the Outrage package format on the Descent 3 expansion
disc) for which the publisher wishes to keep proprietary...they can
write a one-shot archiver without violating the GPL.
MojoSetup is useful for me, and I hope it will be useful to you, too,
under whatever circumstances you use it.
- MojoSetup already targets Unix systems, with the initial grumblings of
support for Mac OS X, Windows and BeOS (!). Other Unixes are trivial to
target. Like the GUI plugins, the platform-specific bits are being
carefully separated out. loki_setup really sort of assumes a Unix (and
in some cases, a Linux) system. Config files are somewhat easier to
reuse between platforms than loki_setup's...even if the most desparate
scenarios, you could just have an if/else block for each platform, since
the config file is a Lua script.
- No autoconf/automake nastiness, we're using CMake (thanks to KDE for
moving to that and making it a viable tool).
- Development is done in a Subversion repository instead of CVS, in the
same way that CMake was a newer, but better alternative to autotools.
- ...probably other things. :)
You can grab MojoSetup from Subversion here:
...and get on the MojoSetup mailing list by sending a blank email to
mojosetup-subscribe at icculus.org ... I figure MojoSetup will be of
serious interest to loki_setup users, which is why I'm posting this
announcement here, but ongoing discussion probably shouldn't happen on
As a test project, I have a build of Duke Nukem 3D for Linux up for
The installer is a 132 kilobyte download, which can then either install
the shareware version by downloading the data at runtime, or install the
full game off a retail disc. The config file for the installer is in
there, too, so you can get an ideal of what rolling your own would be like.
The UI is really rough at the moment and there are some other bugs, too.
There are some obvious features of loki_setup that aren't yet available
in MojoSetup, and the documentation is completely non-existent at this
time...but it gives you an idea of what I'm going for here.
More information about the Lokisetup