If I knew that updating a .plan file was this entertaining, I would have started doing semi-regular updates a long time ago. Oh well. My web space is located at http://icculus.org/~hendersa and I can be reached via e-mail at firstname.lastname@example.org. Archived .plan entries can be seen at http://icculus.org/~hendersa/finger. *********************************** * 11 July 2008 - Out with the old * *********************************** A few months back, I changed jobs. My old company was purchased by a competitor, and I was one of only two people designated as "key personnel" in the acquisition agreement. That meant that aside from the president of the company and myself, everyone else "moved on to new opportunities at other companies". It also meant that I was now on the hook to maintain all of my old company's products. By myself. Sadly, my new company didn't seem to want to learn about our processes, procedures, codebases, toolchains, and general technological trickery that had made us such a strong competitor in the first place. This was all a very great disappointment to me because our competitor had something that my old company did not: manpower. They easily outnumbered us ten to one, yet we were able to take over half of their marketshare. But they couldn't produce anyone capable of taking over the general maintenance of all of our software products. If I were them, I would have taken a few of my high-end engineers and sat them down with the newly acquired assets and, oh, I don't know... learned the skills needed to actually build them and understand how they all worked. Maybe learned how to leverage that technology to make them a stronger company. Used some of our ideas about marketing methods and product differentiation. You know. Get their money's worth. Nope. Not any interest at all. Or not enough interest to put any resources into it. It's disappointing that maybe 20 man-years of high-end embedded software development is getting tossed because its new owners lack the skills to understand how it works. We're talking about getting e-mails with stuff like "I can't find the Visual Studio project file for XXXXXX, so I can't build it for customer XXXXXX. The delivery must be done tomorrow. Please advise." Or "I can't find the module with WinMain(). Please advise." Always with the "Please advise", and also CC'd to everyone and their dog. I wouldn't be too shocked by this usually, except that these were all Unixware and Linux applications built using make! You know that whole concept about having one excellent developer outperforming a whole bunch of mediocre ones? That's not just some theory. It's a fact. And when you are outnumbered by mediocre programmers who are managed by mediocre technical managers, those superstar developers are going to get shot down whenever they make a suggestion. "Why don't we put the Unix software under subversion or CVS for source control?" gets shot down with "Because we currently have everything under VSS, we understand VSS, and we can just check stuff out under Windows and then copy it to a SMB mount that we then copy to an ext3 partition which is available to that Linux machine over there via NFS." Right. Very efficient. I understand this approach if you have support for multiple OSes within a single codebase. In fact, good for you if you have cross- platform functionality within your code. But I don't care if you are married to VSS or not... start thinking of the big picture. That Windows 3.11 crap is hitting end-of-life. I KNOW this. Their CTO SHOULD know this. They should ACT on this. They should grow as a company, continually learning and improving their processes. Also, you might consider the fact that subversion has a Visual Studio plugin, and SVN also works under Linux! Just a little tip, there. I have no doubt that ten years ago, this was the right way to do things. But a funny thing happens when your technical management sits in the same place and does the same thing with the same technology that whole time... they become entrenched in their thinking. Procedures don't change because the current ones are too familiar and convienient. Your competitors start sneaking up on you and snatching your customers because they adapt and move quicker than you can. You can do a few different things to combat these competitors, with "buy them" or "talk smack about them in front of your customers" being real high on this particular company's list. So, we were bought. Then we were thrown into the meat grinder over and over. I'd do deliveries with 40 pieces of software. Twenty of those items were developed by my old company, so I'd be on the hook for those. The other twenty would be items developed by the company that bought us. The new company would have a team similar to the following: two QA, one or two artists, a project manager, three developers, maybe two or three people doing integration on the hardware, and and someone pulling double-duty as a translator to make sure we had proper Chinese, Japanese, French, etc. translations. So you're looking at around ten people who were working on getting twenty software components out the door. I would do the other twenty. The project manager would call me every few hours to ask if it was done yet, and one of the QA would take my software, not be able to figure out how to execute the launch script that I provided, and then send a mail out to everyone and their dog saying "The a.out file was not provided with your upload so I am unable to test. Please advise." When I would try to explain that the software would not run if it was executed on the fileserver, rather than the embedded hardware it was supposed to run on ("I get an error about no framebuffer device being found. I believe your libraries are broken. Please advise."), I'd get a whole lot of frantic handwaving with an explanation like "I am not a coder! I can't do all of these 'telnet' and 'cd' commands of which you speak!" Some of the stuff that I've seen so far: "Your software breaks the entire system. 'ls' no longer works. Please advise." (They copied glibc into the applications directory because they couldn't figure out how a chroot jail environment works, and all the libraries were freaking out.) "There is no audio. Please advise." (The tester had the headset plugged into the wrong piece of hardware.) "Our guy that speaks some Japanese here feels that your professional native- speaking Japanese translator did some stuff wrong." (I'll put my yen on the Japanese guy with the PhD in English literature.) "Your input mechanisms vary from application to application. This is too complex. Please advise." (Our software is not the same thing reskinned over and over and resold as different products. Does every game on the Wii have exactly the same input methods? Oh, it doesn't? Better notify Nintendo that they are doing it all wrong, then.) "Airline X says that game Y needs to be completely redesigned because it is not intuitive. How quickly can you do this?" (Seeing as how that title is damn-near identical to the XBLA version, I'm thinking that the answer is going to be "Never". Sorry Chinese airline... I don't care what you think. I'm going to trust in Microsoft's ability to make user interfaces. The project manager should have enough guts to push back on them for something like this.) "You don't understand! This needs to be done today!" (Right. Which is why you told me at 1100 this morning. The world will not end if someone in the Middle East doesn't get his movie on an airplane. I don't really care if the Arab world believes that the world is their Burger King where they call the shots on every detail of everything. They still aren't getting their embedded version of Bejeweled 2 on such short notice. Your team takes a week to do this and you want one guy to do it in 4 hours? Right.) "I don't have gcc on my system. Where do I get it?" "What is gdb?" "We don't use profilers here. They aren't needed for what we do." "I can't debug on the actual hardware because I can't see the output of the printf calls on the terminal because the framebuffer is up." "When I do that 'ps' thing you told me about, I see ten or so processes with the same name. But I only launched it once. Your launch script is broken." (Holy smokes.) Aside from all that nonsense, I was frequently a victim of "the bus". The core embedded system software (kernel, drivers, system libraries, etc.) was maintained by the hardware manufacturer. The hardware manufacturer had this bad habit of making some change to the system at the last minute before a delivery in a ham-fisted attempt to address some problem. This attempt would inevitably both fail to fix their problem and also break all of our software as well. This made their day, because they would immediately tell the customer that our stuff was broken and that that alone would delay the delivery and that it was all our fault. This fun game of "throw the vendor under the bus" occured on a weekly basis, at least. Someone would threaten to tell the customer that the software would not be released and that we needed to fly someone across the country to be onsite to fix the problem. They would do this to deflect blame and also to force us into diagnosing their problems and sometimes even providing fixes for them. After all, if their stuff is screwed up, then our stuff doesn't get shipped, and we don't get paid. I used to look forward to the challenge of flying out at 0500, getting onsite around lunch, working all night, finding the fix, and then saying "Geeze you guys are dumb" in an e-mail that everyone was copied on before flying back home on a red-eye flight. It was fun roughly twice. After that, it became one of the worst parts of the job. And I did it dozens of times over the years. I started flying out there, sticking lots of workarounds in our stuff so that OUR software worked, and then pointing the finger at the hardware vendor and saying, "No, see, your stuff is broken. Fix it." Then the delivery would fail because the main bugs still remained and we still didn't get paid. After five years, I left the inflight entertainment industry. I'd like to think that I learned some things about how not to treat people and how not to run a business. I developed some great software, but all of that will be buried now that no one has the know-how to maintain it. All of the wiki documentation that I wrote will rust away and be ignored. The clever techniques that we developed will be ignored by entrenched technical management that can't understand them and who refuses to learn about them. Some of the most fun I had at the job was reverse engineering the hardware. The hardware manufacturer would be very secretive about their device drivers and such. So much so that they wouldn't provide the vendors with the information that they needed to write software with any kind of performance. I wasn't about to get put off so easily, so I grabbed a screwdriver and got to work. We'd examine the silkscreening on the chips and then google up chipset documentation to get register information. We'd check out the PCI bus enumerations and then begin memory mapping the pages of memory that held the memory-mapped registers. An mprotect() call would unlock the pages for writing, and then we'd go to town. We were getting the hardware to do all sorts of things that the hardware vendor did not have support for in their device drivers. And we'd do it all in userspace! Video settings, audio configuration, system monitoring... we'd do all of these and more and gain performance benefits that made the other software on the system look like it was standing still. Brutual jobs like that one really take a toll on your health and family life. It was getting to the point where I really needed to move on. I was letting far too much of my life pass me by because I was too busy working. I think that everyone should experience that grind once. Your skills improve dramatically in the right environment because you are learning quickly in a pressure-cooker environment. That was my third "big grind" job. Two of those jobs taught me a lot. The third just gave me ulcers and a lot of frustration. Not surprisingly, that third job was when I was working for the hardware manufacturer that tosses vendors under the bus. I guess there's something to be said for those 9-5 jobs writing SQL queries and spending all day building MFC interfaces in Visual Studio. They are good jobs to have AFTER you've learned your skills and want to do well in a stable environment. As for the stubborn "I only want to do Linux jobs that give me creative freedom" crowd, well... prepare for the big grind that is coming your way. Your employer is going to take that desire and use it against you. Please, please feel free to prove me wrong on this point. Work hard. Learn from those better than you, learn to ignore those that aren't that will mislead you, and learn to distinguish the difference between the two.