[bf1942] 64bit linux BF2 performance stats

[Andreas Fredriksson]

On 7/21/05, James Gurney <james at globalmegacorp.org> wrote:

> Is that still true for NUMA based systems? I'm a little unclear, but my
> understanding was that the main advantage of NUMA was dedicated memory
> per cpu, so migrating from one cpu to the other would lose that
> advantage. Quoting from the (hugely out of date) Linux NUMA page:
(snipped)
>  From that, I'm concluding that locking at least the most memory
> intensive threads to one cpu would be of benefit on a NUMA system, but
> perhaps I'm misunderstanding the overall picture..

You're right, I forgot about NUMA. I'm curious about the cost of
migrating a thread across NUMA CPUs, and especially if the scheduler
still thinks it is worthwhile to do so. In an ideal scenario, the
kernel would correlate the number of page faults (or micro-page faults
over NUMA, whatever they're called) and quickly realize that a number
of threads are competing for shared process memory. These would
perhaps be a bad choice to move to another CPU if one is mostly
I/O-bound. I don't think the default Linux scheduler is that smart
though.

For reference, the main thread of all BF servers (42, V and 2) is very
memory intensive, so it could perhaps benefit from sitting steadily on
one memory node. There is also a thread which pulls in packages from
the network, into memory shared with the main thread. Since these are
cooperating over process memory (and therefore are synchronized), it
might be a real performance loss (in latency, at least, because
mutexes are involved) to switch that thread between NUMA nodes too.

It would be very interesting to see if setting thread affinity would
affect any of
a) total CPU time spent
b) CPU time spent for threads #1 and #2
and also
c) any perceived change in network latency

If anyone has a spare NUMA system and feels up to the task I could
help with looking at the results. I think the pids of the threads can
be gathered from strace, but I'm not familiar with the userland tools
to change the affinity (and I also don't have an Opteron sitting
around).

// Andreas

-- 
And I hate redundancy, and having different functions for the same thing.
        - Linus Torvalds on linux-kernel