No more beta applications, please... (posted 2003-04-25 21:08:01 by icculus):
...we're drowning in beta applications here: several hundred of them, actually.
The America's Army testers have been selected. Don't send any more applications, please. The "betatests" email address now sends a bounce with a polite "we're done, thanks anyhow" message.
Some unofficial, but interesting facts about the applications I got...any conclusions drawn from this are probably incorrect.
- The most popular distro? Gentoo, by quite a margin. Slackware made more of a showing than I expected, too, which suggests that many Linux gamers are also hardcore like-to-recompile-as-much-as-possible types...probably with the same mentality that drives overclockers: every bit of performance tweak they can get is worth the effort. Red Hat, Debian and Mandrake popped up occasionally, not with the frequency I expected. SuSE's presence was almost non existant. There was one mention of Lindows and one of Lycoris, both from the same person. Two or three Linux-From-Scratchers. I don't think I saw any mail from other distros, which means everyone is either rolling their own or using a well-known one. Three people thought this was a Windows beta test.
- Only one person mentioned FreeBSD as his primary platform. Only one other person mentioned it at all.
- The most popular video card? Nvidia, which isn't shocking for a Linux game test at this point. Almost all of those with ATI cards were all high end (Radeon 9000 or better).
- Very few (less than, say, five) people had less than an 800MHz CPU. Maybe less than 20 had less than a 1GHz CPU.
- No one mentioned using the commercial Open Sound System drivers. No one mentioned using Xi Graphics' X server. Very few mentioned that they use ALSA. This suggests to me that the standard, free stuff "just works" or people are too cheap or too lazy to look elsewhere. YMMV.
- A couple people wrote in that are members of the (real) US Army. A couple people wrote in that work for military industry companies like Lockheed Martin. I mention this because they felt it was worth mentioning.
- Overwhelmingly, the beta applications were not from Americans. Many of the people were from Europe (Germany apparently loves this game), others were from South America and Canada, a few from Asia and far Eastern Europe...none of the applications came from countries in the Middle East.
- Maybe 25% had played the game on Win32. Of those, most were still regular players, either through dual boot or another machine. Some were in game clans for ArmyOps. Maybe another 10% mentioned they had tried unsuccessfully to play the game under Wine or WineX. At least half mentioned their only contact with the game was WATCHING other people playing on Win32 (but I suspect this is one of those "I have a friend that has this problem..." stories).
A lot of people asked me what the guidelines for getting accepted into this test were. There wasn't an official algorithm, but here was my thinking process:
I wanted about 30 people (I ended up with 35). Some of these people were in automatically (a QA person from nvidia, a driver developer for 4Front, a distro maintainer, etc), since the game depends on other technology working correctly, it makes sense that some people get access, whether they really test the game or not.
After that, I went through each of the applications and filtered out those that didn't have a chance: those Windows people, those with vastly underpowered hardware, those that didn't tell me what system specs they had at all.
Then, I made a second pass and gave a basic rating to each person...if your rating was high enough, you stayed, if it was too low, you were cut. There isn't a numeric value, really...the same email read twice could rate higher in my mind the first time and lower the second. A few I agonized over.
Things that lowered and highered a given rating, in no particular order:
- How I felt at that moment in time. I was reading these in StarBucks at one point, and spilled my drink on myself. The guy who's application I was reading was cut immediately. Yeah, life's rough.
- Mentioning that you overclock your system (even a little bit) was an immediate rejection. It's just one more thing that can go wrong. PLEASE do not sign up for betas if you overclock your system! Overclocking causes strange bugs that we can not control, even if everything else works peachy keen on your system. Do all game developers a favor on this one, okay?
- Points for humor. Every now and then I'd get an application that just made me laugh, which was nice when going through hundreds of emails that more or less said the same thing.
- Negative points for kissing my butt. "I just want to start off by saying how much I think you rock." got annoying really quickly. I hope I don't sound like an egotistical jerk for saying that.
- Points for having something unique. If everyone is running the same distro/video card/etc, and you show up running something no one else has, that makes you more interesting.
- Negative points for poor grammar and spelling, and using "Netisms" like "lol". This was a rampant problem; very few of the applications I saw would qualify as "well-written"...some of them were downright incoherent.
- Points for having access to variety of hardware to test on.
- Negative points for using a hotmail email address (possible warez monkey).
- Points for giving me something. The guys that mirror some of my bigger stuff to offset my bandwidth usage were shoe-ins.
- Negative points for offering me something. "I've been meaning to donate some hardware to your site, but I definitely will if you put me on this beta" was almost a guaranteed rejection. Preemptive paypal'ing didn't happen this time, but has in the past for other things. I like to pretend I have some amount of integrity, so this doesn't help.
- Negative points for attaching a massive graphic to your email that represents you gaming clan.
- Negative points for sending email in HTML/RTF format if pine didn't know what to do with it.
- Positive points for a low-end (but not TOO low-end!) setup, so I can test the range of hardware.
- Negative points for too little information: some people told me in detail why they would be a good beta tester, but not what kind of computer they have.
- Negative points for too much information: the complete output from the "dmesg" command, what office suites you have installed, the exact model of your hard drive, etc.
- Negative points for WAY too much information: how your therapist told you that ArmyOps was a better outlet for your aggression than hitting your children, etc.
- Negative points for profanity. Cursing to complete strangers, especially when you want something from them, is really stupid...and paints you in a very unflattering light.
- Negative points for slamming Microsoft. I've heard it before, believe me. If all you want to do is fight The Evil Monopoly, then you aren't a good tester. A lot of people had nothing more to say than "I should test this because Microsoft sucks". Well, fine, but I need more, and I need to know you're more than just another zealot.
- Negative points for pleading. It's just a game.
- Points for the right attitude. Expressing that you like the game, you are good at finding bugs, and you are eager to write good bug reports (an art in itself, indeed) made me feel eager to let you in. Expressing your determination well went a long way.
- Points for being in the military. This wasn't an official Army requirement by any means, and I'm not sure why I felt this way, but it DID help in this case.
- Negative points for mentioning that your system runs Quake 3 well. Every bug report I've ever gotten for any other game that was really a driver bug almost certainly got a response of "But this system runs Quake 3 fine, so it must be your bug!". This was totally a flinch reaction on my part.
- Negative points for not mentioning your real name. Double so if the only name I got was some l33tspeak clan handle.
- Negative points for trying to pump me for information: "Hey I want to beta test by the way when is Medal of Honor coming out and is there any chance for a HalfLife port? kthxbye". I hate that.
- Points for being a regular in icculus.org's IRC channel, so I have quick access to you if I need it during the beta test.
- Negative points for being a regular in icculus.org's IRC channel if you're a pain in the ass.
- Positive points for understanding Unix "under the hood".
- Mentioning experience with the Windows version of the game had no real effect on me either way.
- Mentioning that you dual boot didn't phase me, either, assuming you knew what you were doing on a Unix box.
- Working for a Linux and/or gaming news site didn't sway me one way or the other. I don't care if articles are written about the beta test, unless that's the only reason someone was showing up. Ultimately, this is for improving the product and not publicity. Some news site people got in, some did not; I hope they won't consider this an "exclusive scoop" or anything and actually report bugs. :)
Overall, the bottom line was to stand out from the repetition, but not in an annoying way.
After this, I had about 70 people left, and picked out the ones that would bring the best qualities to the table in relation to each other...the goal was to get the widest spread of distros and hardware possible. This was agonizing, and I had to chop some really good people here. After that, I had my beta crew.
I don't know how big companies like Blizzard and EA and such handle their beta tests, but if there's ever a "written portion", I would bet it probably works pretty much the same way. If nothing else, a good portion of these guidelines apply to other aspects of life. Keep them in mind next time you write a term paper for your English class or a cover letter for your resume.
Anyhow, more news on America's Army as I have it.
--icculus.
