If I knew that updating a .plan file was this entertaining, I would have
started doing semi-regular updates a long time ago. Oh well. My web space
is located at http://icculus.org/~hendersa and I can be reached via e-mail
at hendersa@icculus.org.
Archived .plan entries can be seen at http://icculus.org/~hendersa/finger.
***********************************
* 11 July 2008 - Out with the old *
***********************************
A few months back, I changed jobs. My old company was purchased by a competitor,
and I was one of only two people designated as "key personnel" in the acquisition
agreement. That meant that aside from the president of the company and myself,
everyone else "moved on to new opportunities at other companies". It also meant
that I was now on the hook to maintain all of my old company's products. By
myself. Sadly, my new company didn't seem to want to learn about our processes,
procedures, codebases, toolchains, and general technological trickery that had
made us such a strong competitor in the first place.
This was all a very great disappointment to me because our competitor had
something that my old company did not: manpower. They easily outnumbered us ten
to one, yet we were able to take over half of their marketshare. But they
couldn't produce anyone capable of taking over the general maintenance of all of
our software products. If I were them, I would have taken a few of my high-end
engineers and sat them down with the newly acquired assets and, oh, I don't know...
learned the skills needed to actually build them and understand how they all
worked. Maybe learned how to leverage that technology to make them a stronger
company. Used some of our ideas about marketing methods and product
differentiation.
You know. Get their money's worth.
Nope. Not any interest at all. Or not enough interest to put any resources into
it. It's disappointing that maybe 20 man-years of high-end embedded software
development is getting tossed because its new owners lack the skills to understand
how it works. We're talking about getting e-mails with stuff like "I can't find
the Visual Studio project file for XXXXXX, so I can't build it for customer XXXXXX.
The delivery must be done tomorrow. Please advise." Or "I can't find the module
with WinMain(). Please advise." Always with the "Please advise", and also CC'd
to everyone and their dog. I wouldn't be too shocked by this usually, except that
these were all Unixware and Linux applications built using make!
You know that whole concept about having one excellent developer outperforming
a whole bunch of mediocre ones? That's not just some theory. It's a fact. And
when you are outnumbered by mediocre programmers who are managed by mediocre
technical managers, those superstar developers are going to get shot down whenever
they make a suggestion. "Why don't we put the Unix software under subversion or
CVS for source control?" gets shot down with "Because we currently have everything
under VSS, we understand VSS, and we can just check stuff out under Windows and
then copy it to a SMB mount that we then copy to an ext3 partition which is
available to that Linux machine over there via NFS."
Right. Very efficient. I understand this approach if you have support for
multiple OSes within a single codebase. In fact, good for you if you have cross-
platform functionality within your code. But I don't care if you are married to
VSS or not... start thinking of the big picture. That Windows 3.11 crap is hitting
end-of-life. I KNOW this. Their CTO SHOULD know this. They should ACT on this.
They should grow as a company, continually learning and improving their processes.
Also, you might consider the fact that subversion has a Visual Studio plugin, and
SVN also works under Linux! Just a little tip, there.
I have no doubt that ten years ago, this was the right way to do things. But a
funny thing happens when your technical management sits in the same place and does
the same thing with the same technology that whole time... they become entrenched
in their thinking. Procedures don't change because the current ones are too
familiar and convienient. Your competitors start sneaking up on you and snatching
your customers because they adapt and move quicker than you can. You can do a
few different things to combat these competitors, with "buy them" or "talk smack
about them in front of your customers" being real high on this particular
company's list.
So, we were bought. Then we were thrown into the meat grinder over and over. I'd
do deliveries with 40 pieces of software. Twenty of those items were developed
by my old company, so I'd be on the hook for those. The other twenty would be
items developed by the company that bought us. The new company would have a team
similar to the following: two QA, one or two artists, a project manager, three
developers, maybe two or three people doing integration on the hardware, and
and someone pulling double-duty as a translator to make sure we had proper
Chinese, Japanese, French, etc. translations. So you're looking at around ten
people who were working on getting twenty software components out the door.
I would do the other twenty. The project manager would call me every few hours
to ask if it was done yet, and one of the QA would take my software, not be able
to figure out how to execute the launch script that I provided, and then send a
mail out to everyone and their dog saying "The a.out file was not provided with
your upload so I am unable to test. Please advise." When I would try to
explain that the software would not run if it was executed on the fileserver,
rather than the embedded hardware it was supposed to run on ("I get an error
about no framebuffer device being found. I believe your libraries are broken.
Please advise."), I'd get a whole lot of frantic handwaving with an explanation
like "I am not a coder! I can't do all of these 'telnet' and 'cd' commands of
which you speak!"
Some of the stuff that I've seen so far:
"Your software breaks the entire system. 'ls' no longer works. Please advise."
(They copied glibc into the applications directory because they couldn't figure
out how a chroot jail environment works, and all the libraries were freaking out.)
"There is no audio. Please advise."
(The tester had the headset plugged into the wrong piece of hardware.)
"Our guy that speaks some Japanese here feels that your professional native-
speaking Japanese translator did some stuff wrong."
(I'll put my yen on the Japanese guy with the PhD in English literature.)
"Your input mechanisms vary from application to application. This is too complex.
Please advise."
(Our software is not the same thing reskinned over and over and resold as
different products. Does every game on the Wii have exactly the same input methods?
Oh, it doesn't? Better notify Nintendo that they are doing it all wrong, then.)
"Airline X says that game Y needs to be completely redesigned because it is not
intuitive. How quickly can you do this?"
(Seeing as how that title is damn-near identical to the XBLA version, I'm thinking
that the answer is going to be "Never". Sorry Chinese airline... I don't care
what you think. I'm going to trust in Microsoft's ability to make user interfaces.
The project manager should have enough guts to push back on them for something
like this.)
"You don't understand! This needs to be done today!"
(Right. Which is why you told me at 1100 this morning. The world will not end if
someone in the Middle East doesn't get his movie on an airplane. I don't really
care if the Arab world believes that the world is their Burger King where they call
the shots on every detail of everything. They still aren't getting their embedded
version of Bejeweled 2 on such short notice. Your team takes a week to do this and
you want one guy to do it in 4 hours? Right.)
"I don't have gcc on my system. Where do I get it?"
"What is gdb?"
"We don't use profilers here. They aren't needed for what we do."
"I can't debug on the actual hardware because I can't see the output of the printf
calls on the terminal because the framebuffer is up."
"When I do that 'ps' thing you told me about, I see ten or so processes with the
same name. But I only launched it once. Your launch script is broken."
(Holy smokes.)
Aside from all that nonsense, I was frequently a victim of "the bus". The core
embedded system software (kernel, drivers, system libraries, etc.) was maintained
by the hardware manufacturer. The hardware manufacturer had this bad habit of
making some change to the system at the last minute before a delivery in a ham-fisted
attempt to address some problem. This attempt would inevitably both fail to fix their
problem and also break all of our software as well. This made their day, because
they would immediately tell the customer that our stuff was broken and that that alone
would delay the delivery and that it was all our fault. This fun game of "throw the
vendor under the bus" occured on a weekly basis, at least. Someone would threaten
to tell the customer that the software would not be released and that we needed to
fly someone across the country to be onsite to fix the problem. They would do this
to deflect blame and also to force us into diagnosing their problems and sometimes
even providing fixes for them. After all, if their stuff is screwed up, then our
stuff doesn't get shipped, and we don't get paid.
I used to look forward to the challenge of flying out at 0500, getting onsite around
lunch, working all night, finding the fix, and then saying "Geeze you guys are dumb"
in an e-mail that everyone was copied on before flying back home on a red-eye flight.
It was fun roughly twice. After that, it became one of the worst parts of the job.
And I did it dozens of times over the years.
I started flying out there, sticking lots of workarounds in our stuff so that OUR
software worked, and then pointing the finger at the hardware vendor and saying, "No,
see, your stuff is broken. Fix it." Then the delivery would fail because the main
bugs still remained and we still didn't get paid.
After five years, I left the inflight entertainment industry. I'd like to think that
I learned some things about how not to treat people and how not to run a business.
I developed some great software, but all of that will be buried now that no one has
the know-how to maintain it. All of the wiki documentation that I wrote will rust
away and be ignored. The clever techniques that we developed will be ignored by
entrenched technical management that can't understand them and who refuses to learn
about them.
Some of the most fun I had at the job was reverse engineering the hardware. The
hardware manufacturer would be very secretive about their device drivers and such.
So much so that they wouldn't provide the vendors with the information that they
needed to write software with any kind of performance. I wasn't about to get put
off so easily, so I grabbed a screwdriver and got to work. We'd examine the
silkscreening on the chips and then google up chipset documentation to get register
information. We'd check out the PCI bus enumerations and then begin memory mapping
the pages of memory that held the memory-mapped registers. An mprotect() call would
unlock the pages for writing, and then we'd go to town. We were getting the hardware
to do all sorts of things that the hardware vendor did not have support for in their
device drivers. And we'd do it all in userspace! Video settings, audio
configuration, system monitoring... we'd do all of these and more and gain
performance benefits that made the other software on the system look like it was
standing still.
Brutual jobs like that one really take a toll on your health and family life. It
was getting to the point where I really needed to move on. I was letting far too
much of my life pass me by because I was too busy working. I think that everyone
should experience that grind once. Your skills improve dramatically in the right
environment because you are learning quickly in a pressure-cooker environment.
That was my third "big grind" job. Two of those jobs taught me a lot. The third
just gave me ulcers and a lot of frustration. Not surprisingly, that third job
was when I was working for the hardware manufacturer that tosses vendors under the
bus.
I guess there's something to be said for those 9-5 jobs writing SQL queries and
spending all day building MFC interfaces in Visual Studio. They are good jobs to
have AFTER you've learned your skills and want to do well in a stable environment.
As for the stubborn "I only want to do Linux jobs that give me creative freedom"
crowd, well... prepare for the big grind that is coming your way. Your employer
is going to take that desire and use it against you. Please, please feel free to
prove me wrong on this point.
Work hard. Learn from those better than you, learn to ignore those that aren't
that will mislead you, and learn to distinguish the difference between the two.