[lokisetup] Announcing MojoSetup.

Stéphane Peter megastep at megastep.org
Sat May 12 19:35:29 EDT 2007

That's great news. Glad somebody finally took the time to do a well- 
overdue rewrite. I'll check it out and try to bring it up to the  
level of loki_setup for what I need it for, at least (mostly  
supporting a bunch of different flavors of UNIX).

Good job! :)

On May 12, 2007, at 8:44 AM, Ryan C. Gordon wrote:

> ...so I set out to revamp some pieces of loki_setup to make  
> progress on a 2.0 release.
> We first talked about 2.0 in June of 2005:
> http://icculus.org/cgi-bin/ezmlm/ezmlm-cgi?2:mss: 
> 666:200506:fjepgpbjejjcppnfihnf
> And the latest commit to the 2.0 branch happened around that same  
> time:
> http://cvs.icculus.org/cvs/loki_setup/? 
> sortby=date&only_with_tag=SETUP_2_0
> Several of us have been submitting patches to 1.0 in the meantime,  
> making good fixes and adding workarounds for deficiencies but not  
> really making improvements to the codebase.
> Generally loki_setup has worked well enough so it wasn't worth  
> investing the effort to do radical improvements, but there are many  
> many areas where it is either annoying, outdated, or flat out  
> broken. Some of these were issues of lack of engineering resources  
> (we're all busy as hell, and usually desperately patching the  
> installer when issues crop up during crunch time on a product  
> shipping in the next several hours!), and some were things that  
> broke as Linux progressed (glibc incompatibilities, etc). Some were  
> legitimate design decisions that turned out to be less than ideal  
> in the real world. The codebase is very organic, and you can see  
> where various vines of code grew based on concessions made over  
> time. Evolution is a painful process.   :)
> So I started out to do heavy revamping of loki_setup to get to 2.0,  
> starting with the archive plugin stuff:
> http://icculus.org/cgi-bin/ezmlm/ezmlm-cgi?2:mss: 
> 678:200506:nbooliokgneeciafbijn
> ...and as I started to think about the implications of changing  
> that subsystem, I realized that there were other related things I'd  
> like to rework, too, and eventually I realized I wanted to rework  
> almost everything. By the time I mapped out an ideal loki_setup, it  
> wasn't loki_setup at all anymore.
> So I started from scratch.
> I've been making good progress on this (I'm calling it "MojoSetup"  
> at the moment), and while there is still much to be done, I feel  
> it's finally ready to show to the public. Here are some things that  
> panned out pretty well so far...this is a _lot_ of information, but  
> I thought you might be interested in the details, and maybe have  
> some opinions to share.
> Most of MojoSetup was designed explicitly to solve specific  
> shortcomings in loki_setup. It's not that I'm trashing on  
> loki_setup, it's just what I think a nextgen replacement would have  
> to address.
> Some of these are technical notes, some are bullet point features.  
> Skip the ones that bore you.   :)   All of these are currently  
> implemented, but some still need work (the UI could use some  
> polishing, for instance).
> - MojoSetup uses Lua, a lightweight scripting language embedded in  
> the installer, for most of the work: the localization tables are a  
> Lua script, the config file is a Lua script, and a good portion of  
> the actual installer code is Lua script...several pieces of data  
> that the C portions need are stored in Lua tables so they benefit  
> from its garbage collector. I was looking at libxml, which  
> loki_setup uses, and found xml pretty limiting...you have to over- 
> define everything, and do some really awkward things when a quick  
> "if" statement in a script would be way easier for  
> everyone...reference this in the example config file in loki_setup:
>              <option if="+(+(x86,Linux),has-passwd,!false)">
> I don't even know what that does!
> There are lots of other attributes in setup.xml that were clearly  
> special cases that would be better in installer-specific logic,  
> rather than forever locked into the config file schema and bloating  
> the installer itself. Not to mention some attributes needing to be  
> "true", some needing to be "yes", some just needing to exist...the  
> schema and the interpretation of it had its share of problems. A  
> scripting language, even one as liberal as Lua, tends to enforce  
> correct syntax much better.
> Having a scripting language instead of an XML schema lets us keep  
> special cases out of the installer. If you have some strange case  
> (like what the "libc" tag does in loki_setup), you can roll your  
> own test in the config script without bloating the installer:
>           if (MojoSetup.libc == "glibc-2.0") then
>               install_glibc20_bins()   -- or whatever.
>           end
> Also, libxml2 turned out to be _insane_ for file size if you have  
> to statically link it: for example, this test program...
>            http://xmlsoft.org/examples/reader2.c
> ...which parses and validates an XML file and nothing else, is 992  
> kilobytes when statically linked to libxml2 (on PowerPC Mac OS X,  
> compiler optimized for size, no debug symbols). More if you  
> statically link zlib and libiconv, both of which libxml2 requires.  
> Lua as a whole produces a 144k binary that has a full parser,  
> interpreter, and runtime library. If you chop out the parser, it's  
> 120k, and if you strip out the non-essential runtime library bits,  
> it's 72k (and smaller on x86)...and links against the C runtime and  
> nothing else. All those chatty config files can optionally be  
> compiled down to Lua bytecode before shipping, too, to reduce their  
> size.
> Not to mention the general installer logic doesn't need to be fast,  
> but it DOES need some protection from the usual C problems of  
> buffer overruns, uninitialized variables and memory management, so  
> anything that can be reasonably pushed into Lua has been. Calling  
> between C and Lua is, unlike every other scripting language I  
> tried, very very easy, so you can jump back and forth between them  
> as it makes sense to do so.
> Overall, using Lua proved to be a big design win.
> - The installer generally looks for data it needs and wants to  
> install in specific places in a directory tree, but access to that  
> tree is abstracted out, so it can be a real directory or a .zip  
> archive or whatever. This leads to a few interesting scenarios:
> 1) You can have a standard "package format" (filename.mojosetup or  
> whatever), that you could distribute if you expect MojoSetup to be  
> preinstalled on the system. That gets around the executable-bit-on- 
> downloads issue if distributions show up and install  
> MojoSetup...but this isn't required.
> 2) You can put your data in a zip file, and append it to the  
> installer executable. Since the installer doesn't need to unpack  
> itself first to access files like makeself/loki_setup does, you can  
> run the installer and it can open _itself_ like a zipfile, and read  
> from a file hierarchy without writing anything to disk first. This  
> is a win for two reasons: usually loki_setup packages have to be  
> downloaded (one copy to disk), unpacked (two copies, maybe an  
> uncompression stage), run, and THEN the real installation starts  
> (three copies, maybe a second uncompression stage). MojoSetup can  
> avoid all that...zipfiles allow random access, unlike tar files, so  
> it can pretty much start up and respond to the user immediately,  
> and mostly only write files when really installing. At the simplest  
> level, building an installer is just a matter of using a  
> standard .zip utility. You can use other formats besides .zip, this  
> is just the obvious choice in this case.
> 3) Support from installing from archives inside archives. The basic  
> design, by necessity, had to treat a zipfile appended to a binary  
> like a real filesystem that might contain tarballs that need  
> unpacking, etc. Once that abstraction was in place, it cleanly  
> solves a problem I ran into on "Cars: Radiator Springs  
> Adventures" ... it had to use the same install data as the Windows  
> InstallShield installer on the disc...for whatever reason,  
> InstallShield packaged a .zip file of the game data inside a .jar  
> file! The loki_setup installer had a seriously mangled and  
> customized zip.c for this. In MojoSetup, your config would just  
> need to say something like...
>     setup.File {
>         source = "media://retail-disc/install.jar/gamedata.zip";
>     }
> ...and the installer does the Right Thing.
> - Part of the workaround for incompatible and missing GUI libraries in
> loki_setup was to statically link everything you could, and ship
> multiple copies of the installer...a GTK2 version, a statically-linked
> GTK1 version, an ncurses version, etc...there's a whole shell  
> script in
> there to try them all until one actually starts up, since the  
> assumption
> is that the GTK+2 one may be missing a Gnome dependency, etc. It  
> doesn't
> help that each of these version's codebase is mostly a cut-and- 
> paste of
> some other version--including generic installer logic--and bugfixes  
> and
> improvements applied to one don't help the others. It made the  
> download
> big and startup flakey.
> MojoSetup uses plugins for the UI. It has a very tightly-defined  
> interface for what the plugin must do, and keeps all generic logic  
> out of them. While a plugin can be statically linked to the  
> installer (which generally is done for, say, the stdio plugin),  
> most are in their own shared library, which MojoSetup tries to load  
> at startup from inside the installer package. The plugin may fail  
> to load (gtk plugin and there's no GTK+2 libs on the system, etc),  
> and MojoSetup will just go on to the next. Since we don't have to  
> worry about the whole application failing to startup, we never have  
> to statically link a whole GUI toolkit, nor spend effort on trying  
> to dlsym() all the functions we need in GTK+, just in case. Once a  
> plugin loads, it can tell MojoSetup what priority it should be ("I  
> can't connect to an X server, never use me," "I'm a Qt plugin and I  
> can talk to an X server, so I'll work, but you aren't running KDE,  
> so try me last," "I only write to stdout, try me absolutely last,"  
> etc). Users can override the choice of GUI plugins if they like  
> from command line or environment variable. This keeps the download  
> small, lets us provide a bunch of options without worry, and keep  
> the codebase clean.
> When I shipped a Linux version of Candy Cruncher (a casual game  
> from pyrogon.com), I had loki_setup with a statically linked GTK+1  
> UI, and a statically linked curses/dialog UI for the installer,  
> wrapped in a makeself script. I got complaints from dial-up users  
> that the installer's download was bigger than the installed game.  
> In MojoSetup, the GTK+2 UI currently adds a whopping 14 kilobytes  
> to the download _before compression_. The stdio UI is 6. We could  
> probably drop in a KDE, SDL, and wxWidgets implementation for about  
> the same space and let the installer sort out the best plan of  
> attack on the end user's system. Compressed, all these options  
> combined would probably add about 30 to 40 kilobytes to the download.
> - All strings are UTF-8 internally. Unicode and localization were  
> considered from the start. All translations are kept in a single  
> text file (a Lua script, actually), so you don't have to fight with  
> GNU .po files.
> - The UI looks more like Apple's installer (well, it will!). It  
> asks a few basic questions and then does its thing with a "wizard"  
> style interface. It looks more "modern" than loki_setup (well, it  
> will!).
> - There is a "rollback" mechanism. Barring catastrophic disk  
> failure, failed installations can undo everything they wrote to the  
> filesystem, including restoring preexisting files that were  
> overwritten during the install. The installer even makes an earnest  
> attempt at cleanup and rollback if it is crashing with a segfault.
> - As you would expect, the installer can use CD and DVD discs (and  
> USB keychain drives, Samba shares, etc) for installation media, but  
> it can also use data stored on a web server. You can specify files  
> to be obtained at runtime over HTTP or FTP, which MojoSetup will do  
> before starting the actual install loop. This is useful for  
> shipping an extremely small package that gets the user to a UI as  
> fast as possible, then downloads just the optional bits they choose  
> to install...this also allows a vendor to supply updated installer  
> packages on their website without having to repackage the installer  
> itself.
> - MojoSetup is under the zlib license. We had to put a hack in  
> loki_setup for UT2003 to spawn an external application, since we  
> couldn't link the UnrealEngine to loki_setup to verify CD keys  
> without violating the GPL.  This is also useful for installing  
> arbitrary data formats (like the Outrage package format on the  
> Descent 3 expansion disc) for which the publisher wishes to keep  
> proprietary...they can write a one-shot archiver without violating  
> the GPL.
> MojoSetup is useful for me, and I hope it will be useful to you,  
> too, under whatever circumstances you use it.
> - MojoSetup already targets Unix systems, with the initial  
> grumblings of support for Mac OS X, Windows and BeOS (!). Other  
> Unixes are trivial to target. Like the GUI plugins, the platform- 
> specific bits are being carefully separated out. loki_setup really  
> sort of assumes a Unix (and in some cases, a Linux) system. Config  
> files are somewhat easier to reuse between platforms than  
> loki_setup's...even if the most desparate scenarios, you could just  
> have an if/else block for each platform, since the config file is a  
> Lua script.
> - No autoconf/automake nastiness, we're using CMake (thanks to KDE  
> for moving to that and making it a viable tool).
> - Development is done in a Subversion repository instead of CVS, in  
> the same way that CMake was a newer, but better alternative to  
> autotools.
> - ...probably other things.  :)
> You can grab MojoSetup from Subversion here:
>      svn://svn.icculus.org/mojosetup/
> ...and get on the MojoSetup mailing list by sending a blank email  
> to mojosetup-subscribe at icculus.org ... I figure MojoSetup will be  
> of serious interest to loki_setup users, which is why I'm posting  
> this announcement here, but ongoing discussion probably shouldn't  
> happen on this list.
> As a test project, I have a build of Duke Nukem 3D for Linux up for  
> download:
>      http://icculus.org/mojosetup/examples/duke3d/
> The installer is a 132 kilobyte download, which can then either  
> install the shareware version by downloading the data at runtime,  
> or install the full game off a retail disc. The config file for the  
> installer is in there, too, so you can get an ideal of what rolling  
> your own would be like.
> The UI is really rough at the moment and there are some other bugs,  
> too. There are some obvious features of loki_setup that aren't yet  
> available in MojoSetup, and the documentation is completely non- 
> existent at this time...but it gives you an idea of what I'm going  
> for here.
> --ryan.

Stéphane Peter
megastep at megastep.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://icculus.org/pipermail/lokisetup/attachments/20070512/77c9315e/attachment.htm>

More information about the Lokisetup mailing list