[lokisetup] Announcing MojoSetup.

[Jeremy White]

<aol>Me too!</aol>.

This is great to see.

Cheers,

Jeremy

Stéphane Peter wrote:
> That's great news. Glad somebody finally took the time to do a
> well-overdue rewrite. I'll check it out and try to bring it up to the
> level of loki_setup for what I need it for, at least (mostly supporting
> a bunch of different flavors of UNIX).
> 
> Good job! :)
> 
> On May 12, 2007, at 8:44 AM, Ryan C. Gordon wrote:
> 
>>
>> ...so I set out to revamp some pieces of loki_setup to make progress
>> on a 2.0 release.
>>
>> We first talked about 2.0 in June of 2005:
>>
>> http://icculus.org/cgi-bin/ezmlm/ezmlm-cgi?2:mss:666:200506:fjepgpbjejjcppnfihnf
>>
>> And the latest commit to the 2.0 branch happened around that same time:
>>
>> http://cvs.icculus.org/cvs/loki_setup/?sortby=date&only_with_tag=SETUP_2_0
>> <http://cvs.icculus.org/cvs/loki_setup/?sortby=date&only_with_tag=SETUP_2_0>
>>
>> Several of us have been submitting patches to 1.0 in the meantime,
>> making good fixes and adding workarounds for deficiencies but not
>> really making improvements to the codebase.
>>
>> Generally loki_setup has worked well enough so it wasn't worth
>> investing the effort to do radical improvements, but there are many
>> many areas where it is either annoying, outdated, or flat out broken.
>> Some of these were issues of lack of engineering resources (we're all
>> busy as hell, and usually desperately patching the installer when
>> issues crop up during crunch time on a product shipping in the next
>> several hours!), and some were things that broke as Linux progressed
>> (glibc incompatibilities, etc). Some were legitimate design decisions
>> that turned out to be less than ideal in the real world. The codebase
>> is very organic, and you can see where various vines of code grew
>> based on concessions made over time. Evolution is a painful process.   :)
>>
>> So I started out to do heavy revamping of loki_setup to get to 2.0,
>> starting with the archive plugin stuff:
>>
>> http://icculus.org/cgi-bin/ezmlm/ezmlm-cgi?2:mss:678:200506:nbooliokgneeciafbijn
>>
>> ...and as I started to think about the implications of changing that
>> subsystem, I realized that there were other related things I'd like to
>> rework, too, and eventually I realized I wanted to rework almost
>> everything. By the time I mapped out an ideal loki_setup, it wasn't
>> loki_setup at all anymore.
>>
>> So I started from scratch.
>>
>> I've been making good progress on this (I'm calling it "MojoSetup" at
>> the moment), and while there is still much to be done, I feel it's
>> finally ready to show to the public. Here are some things that panned
>> out pretty well so far...this is a _lot_ of information, but I thought
>> you might be interested in the details, and maybe have some opinions
>> to share.
>>
>> Most of MojoSetup was designed explicitly to solve specific
>> shortcomings in loki_setup. It's not that I'm trashing on loki_setup,
>> it's just what I think a nextgen replacement would have to address.
>>
>> Some of these are technical notes, some are bullet point features.
>> Skip the ones that bore you.   :)   All of these are currently
>> implemented, but some still need work (the UI could use some
>> polishing, for instance).
>>
>> - MojoSetup uses Lua, a lightweight scripting language embedded in the
>> installer, for most of the work: the localization tables are a Lua
>> script, the config file is a Lua script, and a good portion of the
>> actual installer code is Lua script...several pieces of data that the
>> C portions need are stored in Lua tables so they benefit from its
>> garbage collector. I was looking at libxml, which loki_setup uses, and
>> found xml pretty limiting...you have to over-define everything, and do
>> some really awkward things when a quick "if" statement in a script
>> would be way easier for everyone...reference this in the example
>> config file in loki_setup:
>>
>>              <option if="+(+(x86,Linux),has-passwd,!false)">
>>
>> I don't even know what that does!
>>
>> There are lots of other attributes in setup.xml that were clearly
>> special cases that would be better in installer-specific logic, rather
>> than forever locked into the config file schema and bloating the
>> installer itself. Not to mention some attributes needing to be "true",
>> some needing to be "yes", some just needing to exist...the schema and
>> the interpretation of it had its share of problems. A scripting
>> language, even one as liberal as Lua, tends to enforce correct syntax
>> much better.
>>
>> Having a scripting language instead of an XML schema lets us keep
>> special cases out of the installer. If you have some strange case
>> (like what the "libc" tag does in loki_setup), you can roll your own
>> test in the config script without bloating the installer:
>>
>>           if (MojoSetup.libc == "glibc-2.0") then
>>               install_glibc20_bins()   -- or whatever.
>>           end
>>
>> Also, libxml2 turned out to be _insane_ for file size if you have to
>> statically link it: for example, this test program...
>>
>>            http://xmlsoft.org/examples/reader2.c
>>
>> ...which parses and validates an XML file and nothing else, is 992
>> kilobytes when statically linked to libxml2 (on PowerPC Mac OS X,
>> compiler optimized for size, no debug symbols). More if you statically
>> link zlib and libiconv, both of which libxml2 requires. Lua as a whole
>> produces a 144k binary that has a full parser, interpreter, and
>> runtime library. If you chop out the parser, it's 120k, and if you
>> strip out the non-essential runtime library bits, it's 72k (and
>> smaller on x86)...and links against the C runtime and nothing else.
>> All those chatty config files can optionally be compiled down to Lua
>> bytecode before shipping, too, to reduce their size.
>>
>> Not to mention the general installer logic doesn't need to be fast,
>> but it DOES need some protection from the usual C problems of buffer
>> overruns, uninitialized variables and memory management, so anything
>> that can be reasonably pushed into Lua has been. Calling between C and
>> Lua is, unlike every other scripting language I tried, very very easy,
>> so you can jump back and forth between them as it makes sense to do so.
>>
>> Overall, using Lua proved to be a big design win.
>>
>>
>> - The installer generally looks for data it needs and wants to install
>> in specific places in a directory tree, but access to that tree is
>> abstracted out, so it can be a real directory or a .zip archive or
>> whatever. This leads to a few interesting scenarios:
>>
>> 1) You can have a standard "package format" (filename.mojosetup or
>> whatever), that you could distribute if you expect MojoSetup to be
>> preinstalled on the system. That gets around the
>> executable-bit-on-downloads issue if distributions show up and install
>> MojoSetup...but this isn't required.
>>
>> 2) You can put your data in a zip file, and append it to the installer
>> executable. Since the installer doesn't need to unpack itself first to
>> access files like makeself/loki_setup does, you can run the installer
>> and it can open _itself_ like a zipfile, and read from a file
>> hierarchy without writing anything to disk first. This is a win for
>> two reasons: usually loki_setup packages have to be downloaded (one
>> copy to disk), unpacked (two copies, maybe an uncompression stage),
>> run, and THEN the real installation starts (three copies, maybe a
>> second uncompression stage). MojoSetup can avoid all that...zipfiles
>> allow random access, unlike tar files, so it can pretty much start up
>> and respond to the user immediately, and mostly only write files when
>> really installing. At the simplest level, building an installer is
>> just a matter of using a standard .zip utility. You can use other
>> formats besides .zip, this is just the obvious choice in this case.
>>
>> 3) Support from installing from archives inside archives. The basic
>> design, by necessity, had to treat a zipfile appended to a binary like
>> a real filesystem that might contain tarballs that need unpacking,
>> etc. Once that abstraction was in place, it cleanly solves a problem I
>> ran into on "Cars: Radiator Springs Adventures" ... it had to use the
>> same install data as the Windows InstallShield installer on the
>> disc...for whatever reason, InstallShield packaged a .zip file of the
>> game data inside a .jar file! The loki_setup installer had a seriously
>> mangled and customized zip.c for this. In MojoSetup, your config would
>> just need to say something like...
>>
>>     setup.File {
>>         source = "media://retail-disc/install.jar/gamedata.zip";
>>     }
>>
>> ...and the installer does the Right Thing.
>>
>>
>> - Part of the workaround for incompatible and missing GUI libraries in
>> loki_setup was to statically link everything you could, and ship
>> multiple copies of the installer...a GTK2 version, a statically-linked
>> GTK1 version, an ncurses version, etc...there's a whole shell script in
>> there to try them all until one actually starts up, since the assumption
>> is that the GTK+2 one may be missing a Gnome dependency, etc. It doesn't
>> help that each of these version's codebase is mostly a cut-and-paste of
>> some other version--including generic installer logic--and bugfixes and
>> improvements applied to one don't help the others. It made the download
>> big and startup flakey.
>>
>> MojoSetup uses plugins for the UI. It has a very tightly-defined
>> interface for what the plugin must do, and keeps all generic logic out
>> of them. While a plugin can be statically linked to the installer
>> (which generally is done for, say, the stdio plugin), most are in
>> their own shared library, which MojoSetup tries to load at startup
>> from inside the installer package. The plugin may fail to load (gtk
>> plugin and there's no GTK+2 libs on the system, etc), and MojoSetup
>> will just go on to the next. Since we don't have to worry about the
>> whole application failing to startup, we never have to statically link
>> a whole GUI toolkit, nor spend effort on trying to dlsym() all the
>> functions we need in GTK+, just in case. Once a plugin loads, it can
>> tell MojoSetup what priority it should be ("I can't connect to an X
>> server, never use me," "I'm a Qt plugin and I can talk to an X server,
>> so I'll work, but you aren't running KDE, so try me last," "I only
>> write to stdout, try me absolutely last," etc). Users can override the
>> choice of GUI plugins if they like from command line or environment
>> variable. This keeps the download small, lets us provide a bunch of
>> options without worry, and keep the codebase clean.
>>
>> When I shipped a Linux version of Candy Cruncher (a casual game from
>> pyrogon.com), I had loki_setup with a statically linked GTK+1 UI, and
>> a statically linked curses/dialog UI for the installer, wrapped in a
>> makeself script. I got complaints from dial-up users that the
>> installer's download was bigger than the installed game. In MojoSetup,
>> the GTK+2 UI currently adds a whopping 14 kilobytes to the download
>> _before compression_. The stdio UI is 6. We could probably drop in a
>> KDE, SDL, and wxWidgets implementation for about the same space and
>> let the installer sort out the best plan of attack on the end user's
>> system. Compressed, all these options combined would probably add
>> about 30 to 40 kilobytes to the download.
>>
>>
>> - All strings are UTF-8 internally. Unicode and localization were
>> considered from the start. All translations are kept in a single text
>> file (a Lua script, actually), so you don't have to fight with GNU .po
>> files.
>>
>>
>> - The UI looks more like Apple's installer (well, it will!). It asks a
>> few basic questions and then does its thing with a "wizard" style
>> interface. It looks more "modern" than loki_setup (well, it will!).
>>
>>
>> - There is a "rollback" mechanism. Barring catastrophic disk failure,
>> failed installations can undo everything they wrote to the filesystem,
>> including restoring preexisting files that were overwritten during the
>> install. The installer even makes an earnest attempt at cleanup and
>> rollback if it is crashing with a segfault.
>>
>>
>> - As you would expect, the installer can use CD and DVD discs (and USB
>> keychain drives, Samba shares, etc) for installation media, but it can
>> also use data stored on a web server. You can specify files to be
>> obtained at runtime over HTTP or FTP, which MojoSetup will do before
>> starting the actual install loop. This is useful for shipping an
>> extremely small package that gets the user to a UI as fast as
>> possible, then downloads just the optional bits they choose to
>> install...this also allows a vendor to supply updated installer
>> packages on their website without having to repackage the installer
>> itself.
>>
>>
>> - MojoSetup is under the zlib license. We had to put a hack in
>> loki_setup for UT2003 to spawn an external application, since we
>> couldn't link the UnrealEngine to loki_setup to verify CD keys without
>> violating the GPL.  This is also useful for installing arbitrary data
>> formats (like the Outrage package format on the Descent 3 expansion
>> disc) for which the publisher wishes to keep proprietary...they can
>> write a one-shot archiver without violating the GPL.
>>
>> MojoSetup is useful for me, and I hope it will be useful to you, too,
>> under whatever circumstances you use it.
>>
>>
>> - MojoSetup already targets Unix systems, with the initial grumblings
>> of support for Mac OS X, Windows and BeOS (!). Other Unixes are
>> trivial to target. Like the GUI plugins, the platform-specific bits
>> are being carefully separated out. loki_setup really sort of assumes a
>> Unix (and in some cases, a Linux) system. Config files are somewhat
>> easier to reuse between platforms than loki_setup's...even if the most
>> desparate scenarios, you could just have an if/else block for each
>> platform, since the config file is a Lua script.
>>
>>
>> - No autoconf/automake nastiness, we're using CMake (thanks to KDE for
>> moving to that and making it a viable tool).
>>
>>
>> - Development is done in a Subversion repository instead of CVS, in
>> the same way that CMake was a newer, but better alternative to autotools.
>>
>>
>> - ...probably other things.  :)
>>
>>
>> You can grab MojoSetup from Subversion here:
>>
>>      svn://svn.icculus.org/mojosetup/
>>
>>
>> ...and get on the MojoSetup mailing list by sending a blank email to
>> mojosetup-subscribe at icculus.org
>> <mailto:mojosetup-subscribe at icculus.org> ... I figure MojoSetup will
>> be of serious interest to loki_setup users, which is why I'm posting
>> this announcement here, but ongoing discussion probably shouldn't
>> happen on this list.
>>
>> As a test project, I have a build of Duke Nukem 3D for Linux up for
>> download:
>>
>>      http://icculus.org/mojosetup/examples/duke3d/
>>
>> The installer is a 132 kilobyte download, which can then either
>> install the shareware version by downloading the data at runtime, or
>> install the full game off a retail disc. The config file for the
>> installer is in there, too, so you can get an ideal of what rolling
>> your own would be like.
>>
>> The UI is really rough at the moment and there are some other bugs,
>> too. There are some obvious features of loki_setup that aren't yet
>> available in MojoSetup, and the documentation is completely
>> non-existent at this time...but it gives you an idea of what I'm going
>> for here.
>>
>> --ryan.
>>
>>
> 
> --
> Stéphane Peter
> megastep at megastep.org <mailto:megastep at megastep.org>
> 
> 
>