[lokisetup] Announcing MojoSetup.
Stéphane Peter
megastep at megastep.org
Sat May 12 19:35:29 EDT 2007
That's great news. Glad somebody finally took the time to do a well-
overdue rewrite. I'll check it out and try to bring it up to the
level of loki_setup for what I need it for, at least (mostly
supporting a bunch of different flavors of UNIX).
Good job! :)
On May 12, 2007, at 8:44 AM, Ryan C. Gordon wrote:
>
> ...so I set out to revamp some pieces of loki_setup to make
> progress on a 2.0 release.
>
> We first talked about 2.0 in June of 2005:
>
> http://icculus.org/cgi-bin/ezmlm/ezmlm-cgi?2:mss:
> 666:200506:fjepgpbjejjcppnfihnf
>
> And the latest commit to the 2.0 branch happened around that same
> time:
>
> http://cvs.icculus.org/cvs/loki_setup/?
> sortby=date&only_with_tag=SETUP_2_0
>
> Several of us have been submitting patches to 1.0 in the meantime,
> making good fixes and adding workarounds for deficiencies but not
> really making improvements to the codebase.
>
> Generally loki_setup has worked well enough so it wasn't worth
> investing the effort to do radical improvements, but there are many
> many areas where it is either annoying, outdated, or flat out
> broken. Some of these were issues of lack of engineering resources
> (we're all busy as hell, and usually desperately patching the
> installer when issues crop up during crunch time on a product
> shipping in the next several hours!), and some were things that
> broke as Linux progressed (glibc incompatibilities, etc). Some were
> legitimate design decisions that turned out to be less than ideal
> in the real world. The codebase is very organic, and you can see
> where various vines of code grew based on concessions made over
> time. Evolution is a painful process. :)
>
> So I started out to do heavy revamping of loki_setup to get to 2.0,
> starting with the archive plugin stuff:
>
> http://icculus.org/cgi-bin/ezmlm/ezmlm-cgi?2:mss:
> 678:200506:nbooliokgneeciafbijn
>
> ...and as I started to think about the implications of changing
> that subsystem, I realized that there were other related things I'd
> like to rework, too, and eventually I realized I wanted to rework
> almost everything. By the time I mapped out an ideal loki_setup, it
> wasn't loki_setup at all anymore.
>
> So I started from scratch.
>
> I've been making good progress on this (I'm calling it "MojoSetup"
> at the moment), and while there is still much to be done, I feel
> it's finally ready to show to the public. Here are some things that
> panned out pretty well so far...this is a _lot_ of information, but
> I thought you might be interested in the details, and maybe have
> some opinions to share.
>
> Most of MojoSetup was designed explicitly to solve specific
> shortcomings in loki_setup. It's not that I'm trashing on
> loki_setup, it's just what I think a nextgen replacement would have
> to address.
>
> Some of these are technical notes, some are bullet point features.
> Skip the ones that bore you. :) All of these are currently
> implemented, but some still need work (the UI could use some
> polishing, for instance).
>
> - MojoSetup uses Lua, a lightweight scripting language embedded in
> the installer, for most of the work: the localization tables are a
> Lua script, the config file is a Lua script, and a good portion of
> the actual installer code is Lua script...several pieces of data
> that the C portions need are stored in Lua tables so they benefit
> from its garbage collector. I was looking at libxml, which
> loki_setup uses, and found xml pretty limiting...you have to over-
> define everything, and do some really awkward things when a quick
> "if" statement in a script would be way easier for
> everyone...reference this in the example config file in loki_setup:
>
> <option if="+(+(x86,Linux),has-passwd,!false)">
>
> I don't even know what that does!
>
> There are lots of other attributes in setup.xml that were clearly
> special cases that would be better in installer-specific logic,
> rather than forever locked into the config file schema and bloating
> the installer itself. Not to mention some attributes needing to be
> "true", some needing to be "yes", some just needing to exist...the
> schema and the interpretation of it had its share of problems. A
> scripting language, even one as liberal as Lua, tends to enforce
> correct syntax much better.
>
> Having a scripting language instead of an XML schema lets us keep
> special cases out of the installer. If you have some strange case
> (like what the "libc" tag does in loki_setup), you can roll your
> own test in the config script without bloating the installer:
>
> if (MojoSetup.libc == "glibc-2.0") then
> install_glibc20_bins() -- or whatever.
> end
>
> Also, libxml2 turned out to be _insane_ for file size if you have
> to statically link it: for example, this test program...
>
> http://xmlsoft.org/examples/reader2.c
>
> ...which parses and validates an XML file and nothing else, is 992
> kilobytes when statically linked to libxml2 (on PowerPC Mac OS X,
> compiler optimized for size, no debug symbols). More if you
> statically link zlib and libiconv, both of which libxml2 requires.
> Lua as a whole produces a 144k binary that has a full parser,
> interpreter, and runtime library. If you chop out the parser, it's
> 120k, and if you strip out the non-essential runtime library bits,
> it's 72k (and smaller on x86)...and links against the C runtime and
> nothing else. All those chatty config files can optionally be
> compiled down to Lua bytecode before shipping, too, to reduce their
> size.
>
> Not to mention the general installer logic doesn't need to be fast,
> but it DOES need some protection from the usual C problems of
> buffer overruns, uninitialized variables and memory management, so
> anything that can be reasonably pushed into Lua has been. Calling
> between C and Lua is, unlike every other scripting language I
> tried, very very easy, so you can jump back and forth between them
> as it makes sense to do so.
>
> Overall, using Lua proved to be a big design win.
>
>
> - The installer generally looks for data it needs and wants to
> install in specific places in a directory tree, but access to that
> tree is abstracted out, so it can be a real directory or a .zip
> archive or whatever. This leads to a few interesting scenarios:
>
> 1) You can have a standard "package format" (filename.mojosetup or
> whatever), that you could distribute if you expect MojoSetup to be
> preinstalled on the system. That gets around the executable-bit-on-
> downloads issue if distributions show up and install
> MojoSetup...but this isn't required.
>
> 2) You can put your data in a zip file, and append it to the
> installer executable. Since the installer doesn't need to unpack
> itself first to access files like makeself/loki_setup does, you can
> run the installer and it can open _itself_ like a zipfile, and read
> from a file hierarchy without writing anything to disk first. This
> is a win for two reasons: usually loki_setup packages have to be
> downloaded (one copy to disk), unpacked (two copies, maybe an
> uncompression stage), run, and THEN the real installation starts
> (three copies, maybe a second uncompression stage). MojoSetup can
> avoid all that...zipfiles allow random access, unlike tar files, so
> it can pretty much start up and respond to the user immediately,
> and mostly only write files when really installing. At the simplest
> level, building an installer is just a matter of using a
> standard .zip utility. You can use other formats besides .zip, this
> is just the obvious choice in this case.
>
> 3) Support from installing from archives inside archives. The basic
> design, by necessity, had to treat a zipfile appended to a binary
> like a real filesystem that might contain tarballs that need
> unpacking, etc. Once that abstraction was in place, it cleanly
> solves a problem I ran into on "Cars: Radiator Springs
> Adventures" ... it had to use the same install data as the Windows
> InstallShield installer on the disc...for whatever reason,
> InstallShield packaged a .zip file of the game data inside a .jar
> file! The loki_setup installer had a seriously mangled and
> customized zip.c for this. In MojoSetup, your config would just
> need to say something like...
>
> setup.File {
> source = "media://retail-disc/install.jar/gamedata.zip";
> }
>
> ...and the installer does the Right Thing.
>
>
> - Part of the workaround for incompatible and missing GUI libraries in
> loki_setup was to statically link everything you could, and ship
> multiple copies of the installer...a GTK2 version, a statically-linked
> GTK1 version, an ncurses version, etc...there's a whole shell
> script in
> there to try them all until one actually starts up, since the
> assumption
> is that the GTK+2 one may be missing a Gnome dependency, etc. It
> doesn't
> help that each of these version's codebase is mostly a cut-and-
> paste of
> some other version--including generic installer logic--and bugfixes
> and
> improvements applied to one don't help the others. It made the
> download
> big and startup flakey.
>
> MojoSetup uses plugins for the UI. It has a very tightly-defined
> interface for what the plugin must do, and keeps all generic logic
> out of them. While a plugin can be statically linked to the
> installer (which generally is done for, say, the stdio plugin),
> most are in their own shared library, which MojoSetup tries to load
> at startup from inside the installer package. The plugin may fail
> to load (gtk plugin and there's no GTK+2 libs on the system, etc),
> and MojoSetup will just go on to the next. Since we don't have to
> worry about the whole application failing to startup, we never have
> to statically link a whole GUI toolkit, nor spend effort on trying
> to dlsym() all the functions we need in GTK+, just in case. Once a
> plugin loads, it can tell MojoSetup what priority it should be ("I
> can't connect to an X server, never use me," "I'm a Qt plugin and I
> can talk to an X server, so I'll work, but you aren't running KDE,
> so try me last," "I only write to stdout, try me absolutely last,"
> etc). Users can override the choice of GUI plugins if they like
> from command line or environment variable. This keeps the download
> small, lets us provide a bunch of options without worry, and keep
> the codebase clean.
>
> When I shipped a Linux version of Candy Cruncher (a casual game
> from pyrogon.com), I had loki_setup with a statically linked GTK+1
> UI, and a statically linked curses/dialog UI for the installer,
> wrapped in a makeself script. I got complaints from dial-up users
> that the installer's download was bigger than the installed game.
> In MojoSetup, the GTK+2 UI currently adds a whopping 14 kilobytes
> to the download _before compression_. The stdio UI is 6. We could
> probably drop in a KDE, SDL, and wxWidgets implementation for about
> the same space and let the installer sort out the best plan of
> attack on the end user's system. Compressed, all these options
> combined would probably add about 30 to 40 kilobytes to the download.
>
>
> - All strings are UTF-8 internally. Unicode and localization were
> considered from the start. All translations are kept in a single
> text file (a Lua script, actually), so you don't have to fight with
> GNU .po files.
>
>
> - The UI looks more like Apple's installer (well, it will!). It
> asks a few basic questions and then does its thing with a "wizard"
> style interface. It looks more "modern" than loki_setup (well, it
> will!).
>
>
> - There is a "rollback" mechanism. Barring catastrophic disk
> failure, failed installations can undo everything they wrote to the
> filesystem, including restoring preexisting files that were
> overwritten during the install. The installer even makes an earnest
> attempt at cleanup and rollback if it is crashing with a segfault.
>
>
> - As you would expect, the installer can use CD and DVD discs (and
> USB keychain drives, Samba shares, etc) for installation media, but
> it can also use data stored on a web server. You can specify files
> to be obtained at runtime over HTTP or FTP, which MojoSetup will do
> before starting the actual install loop. This is useful for
> shipping an extremely small package that gets the user to a UI as
> fast as possible, then downloads just the optional bits they choose
> to install...this also allows a vendor to supply updated installer
> packages on their website without having to repackage the installer
> itself.
>
>
> - MojoSetup is under the zlib license. We had to put a hack in
> loki_setup for UT2003 to spawn an external application, since we
> couldn't link the UnrealEngine to loki_setup to verify CD keys
> without violating the GPL. This is also useful for installing
> arbitrary data formats (like the Outrage package format on the
> Descent 3 expansion disc) for which the publisher wishes to keep
> proprietary...they can write a one-shot archiver without violating
> the GPL.
>
> MojoSetup is useful for me, and I hope it will be useful to you,
> too, under whatever circumstances you use it.
>
>
> - MojoSetup already targets Unix systems, with the initial
> grumblings of support for Mac OS X, Windows and BeOS (!). Other
> Unixes are trivial to target. Like the GUI plugins, the platform-
> specific bits are being carefully separated out. loki_setup really
> sort of assumes a Unix (and in some cases, a Linux) system. Config
> files are somewhat easier to reuse between platforms than
> loki_setup's...even if the most desparate scenarios, you could just
> have an if/else block for each platform, since the config file is a
> Lua script.
>
>
> - No autoconf/automake nastiness, we're using CMake (thanks to KDE
> for moving to that and making it a viable tool).
>
>
> - Development is done in a Subversion repository instead of CVS, in
> the same way that CMake was a newer, but better alternative to
> autotools.
>
>
> - ...probably other things. :)
>
>
> You can grab MojoSetup from Subversion here:
>
> svn://svn.icculus.org/mojosetup/
>
>
> ...and get on the MojoSetup mailing list by sending a blank email
> to mojosetup-subscribe at icculus.org ... I figure MojoSetup will be
> of serious interest to loki_setup users, which is why I'm posting
> this announcement here, but ongoing discussion probably shouldn't
> happen on this list.
>
> As a test project, I have a build of Duke Nukem 3D for Linux up for
> download:
>
> http://icculus.org/mojosetup/examples/duke3d/
>
> The installer is a 132 kilobyte download, which can then either
> install the shareware version by downloading the data at runtime,
> or install the full game off a retail disc. The config file for the
> installer is in there, too, so you can get an ideal of what rolling
> your own would be like.
>
> The UI is really rough at the moment and there are some other bugs,
> too. There are some obvious features of loki_setup that aren't yet
> available in MojoSetup, and the documentation is completely non-
> existent at this time...but it gives you an idea of what I'm going
> for here.
>
> --ryan.
>
>
--
Stéphane Peter
megastep at megastep.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://icculus.org/pipermail/lokisetup/attachments/20070512/77c9315e/attachment.htm>
More information about the Lokisetup
mailing list