<html>
<body>
<br>
Dave,<br><br>
This is most useful! No apologies needed for length. Thanks for
taking the time!<br>
The single install has some added significant impact as well... I presume
it is<br>
sticky bit, hence same shared code segments, saves memory big time,
as<br>
well as load time for each instance.<br><br>
The max rate is as you pointed out specific to the users
experience. Your<br>
findings, per game is most useful information for all running a multiple
gaming<br>
server out there.<br><br>
The Nice values.. that is a bit of quagmire. The information is very
useful, but<br>
as you note, very specific to the servers you are running and the load
indeed.<br><br>
There is no point of running SOF2 at 15K if 8K will do! What you
have brought<br>
out, that based on any other comments, others are already doing and I
am<br>
completely in the dark, or something very significant. I believe I am not
in<br>
the dark, and your hard work can help others greatly.<br><br>
What Dave has done here is:<br>
1. Determined the max_rate point of diminishing returns on a game by game
basis.<br>
(i.e. Don't run em at 25K if they see no difference
above 15K, or 8K.<br>
These are changes you can make today in your configs,
and you will not<br>
impact your quality, but will bring down your
bandwidth use, and a side effect<br>
which I will mention in a moment.<br><br>
2. The Nice, based on a quantified set of "server type".
Nice being relative,<br>
it will need to be tweaked for each server, but the
general solution above is<br>
sound based on how nice works.<br><br>
3. Another point, that is not as apparent, mentioned, but is real in the
relativeness<br>
of the Linux Kernel is that processes are in one of
three states... (Assuming the<br>
simple case of non-multi thread for simplification). A
process in gaming is in 4 states:<br>
1. Idle (Ha!) not likely but indeed you may find
lagging servers, and 40% Idle time.<br>
(Not withstanding the
reporting tools, can calculation deficiencies)<br>
2. Waiting in the CPU allocation queue for CPU cycles
to execute<br>
3. Executing (run)<br>
4. In I/O Wait state<br><br>
NICE will impact the time the CPU is allotted to a
process, as well as it's relative priority<br>
to attain the CPU. But since all useful programs
when running move between the state<br>
of CPU wait, CPU Run, I/O wait, I/O Complete, CPU Wait
(and the DFA scheduling starts<br>
again), one can see a few things that are
important:<br><br>
1. The amount of CPU it will require is specific to
many of the games attributes, but it all<br>
boils down to
transmitting/receiving the I/O it waited for or received.<br>
2. Hence, if you decrease the Max_rate, less I/O Wait
states, makes the process available<br>
for CPU execution with a
higher frequency. This is where the NICE value becomes<br>
relative to the game's CPU
requirements AND the I/O process that is not always clear.<br><br>
Ultimately, the best use of resources can be had by
finding the max_rate for user quality.<br>
This will define the I/O wait, and has a more farther
reaching than simply bandwidth utilization.<br>
When niced properly for it's server type, the
bandwidth per user is as efficient as it can get.<br>
The CPU Run time allocation, if you nice it down too
much relatively to your other gaming servers<br>
will experience lag. Ironically not
computational, but in kernel space setting up DMA transfers<br>
of I/O related data from it's previous state in
the DFA.<br><br>
His work is significant. I don't know if he knows all the inner workings
of the kernel, I suspect<br>
so, as this does not happen by accident. I will tell you that other
server companies that<br>
have worked out these relationships have not shared them for competitive
advantage.<br><br>
There is a lot of useful information provided here by Dave, and he
wrapped up some complexities<br>
very nicely in to a nice general package for server admins to work with.
I just tried to place my<br>
thoughts about what he is really doing in the kernel in perspective to
help anyone that wants to<br>
take the "fine tune" process to a more granular level when
troubleshooting two or more servers<br>
that may have lag issues, giving you an idea of the basic DFA scheduler,
so with a peek with<br>
tools like vmstat, top, you can get an idea of what best to tweak to try
to resolve your issue.<br><br>
At the same time, the info Dave provided should give beginning admins
that have these<br>
powerful servers, but may not be able to seem to get all they wish to get
out to the server.<br><br>
My thanks Dave for the information and your time.<br><br>
I noticed your CPU is very above 25% on a 2.4 Dual Pent Proc, with 10Mbs
FD. Very Nice!<br><br>
My many thanks.<br><br>
Dr. D<br><br>
At 09:57 AM 12/3/2004, you wrote:<br>
<blockquote type=cite class=cite cite>You can see the overview at the CS
client page
<a href="http://www.clanservers.net/clients.asp">http://www.clanservers.net/clients.asp</a>.
We're in the second group "Game Servers by the Box at
L33tgames." We have a rather special relationship with CS because
we've been with them since the beginning and we do a lot of beta work for
them. Except for initial install we maintain all our game servers
ourselves (which is rather unique in the game server business) via
telnet. Each game has it's own user account and the game itself is
running in its own Screen. We use Apache to serve up player stats and
overall stats.<br>
<br>
You can see our specific server stats at
<a href="http://washington.clanservers.net/">http://washington.clanservers.net/</a>.<br>
<br>
As for the tuning. Indeed we have spent a lot of time tuning specific
apps and the 15k vs. 25k max_rates per client are a tradeoff between
individual player performance and overall system performance. We were
encountering the same sort of cross-server interference (lag) problems
that you mentioned. It was solved through a combination of nice settings
(match servers -15, public servers -10, custom map servers -5, telnet
sessions 0) and the per seat throttling. The throttling is different for
each game type because WET for instance chokes at 15k but is quite
responsive at 25k. SOF2 on the other hand is tolerable at 8k. Each game
will use the full per player BW limit. The upper limit is of course the
total BW limit set by the connect contract. When we blow our monthly
allotment, which we have done when all games were running full BW, the
connect overcharge is a killer. The decision to use one IP per game with
consecutive port numbers for instances is somewhat arbitrary, however, we
do run from a single master install per game and symlink all common files
into the instances. There is really only one COD:UO "install"
with 3 symlinked instances. That way we only patch in one place per
game.<br>
<br>
In fact all servers are running simultaneously (plus a 16 man SOF2 server
we sublet) and we rarely encounter any cross-game lag anymore. Of course
the match servers are not normally populated but the others are. We have
had AA, BF1942 and BF:V servers in the mix at one time or another as
well.<br>
<br>
Wheu, this was rather long winded, but I hope it helps.<br>
Dave<br><br>
-------- Original Message --------<br>
Subject: RE: [cod] Multiple service server + Going off list due to <br>
"spamming"<br>
From: "Mark J. DeFilippis" <defilm@acm.org><br>
Date: Fri, December 03, 2004 4:46 am<br>
To: cod@icculus.org<br><br>
<br>
What nifty utility are U using to format that nice output list<br>
or is that a flat file, or you use command line args with a<br>
ps -el | <perl script, sed, awk, etc>?<br><br>
<br>
On a Dual Pent 4, 3.2Ghz HT, 2GB Ram, 100Mbs, (6 IP's) when my<br>
UO foy 24x7 fills up with 32 users (No limits on tanks or etc. It<br>
is Ladder compliant), ONE cpu runs at 60-80% (Combined Real<br>
and Virtual).<br><br>
I have 24 Man Rifles Only server running, and a BF1942 server<br>
32 user. The UO Rifles only will Lag if the BF1942 server and<br>
32 user FOY fill up. Of course they get really ticked off in the<br>
rifles only server, as the most skill is indeed required there...<br>
(You have a bolt action rifle, or you can bash the guy. No scopes,<br>
smg's, nades (except smoke)). So if they get lag, it can be<br>
pretty tough to hit a target, and becomes VERY frustrating<br>
quickly..<br><br>
I presume all these are not actually running at the same time?<br>
Also, I noticed some of your Max_rates are at 25K, some 15K.<br>
I never noticed much of a difference between 15K and 25K? But<br>
then again, I have not run as many virtual servers as you are.<br>
I ask, because if you set them lower, I presume you did so for<br>
a reason, or computed them and hence 15K vs default 25K?<br><br>
If you don't mind, can you tell us about this server?<br><br>
Physical... <br>
Your normal utilization? Peak utilization?<br>
Limiting Mods?<br><br>
I think the information could be insightful, certainly for me<br>
simply by the fact that when the UO server is full and the<br>
BF1942 are full, there is hardly any CPU left...<br><br>
Thanks<br><br>
Mark<br><br>
<br><br>
At 11:54 PM 12/2/2004, you wrote:<blockquote type=cite class=cite cite>
<dl>
<dd>You can do a similar setup using virtual IPs and assigning each game
its own
<dd>IP. Of course you need to "own" the IP you are
assigning. If you only have
<dd>access to one IP then assigning the games different ports is the way
to go.<br>
<dd>Donovan<br>
<dd>-----Original Message-----
<dd>From: David A. Fuess
[<a href="mailto:david@fuess.net" eudora="autourl">mailto:david@fuess.net</a>]
<dd>Sent: Thursday, December 02, 2004 2:27 PM
<dd>To: cod@icculus.org
<dd>Subject: Re: [cod] Multiple service server + Going off list due to
<dd>"spamming"<br>
<dd>I run 3 UO servers on one machine along with several other game
types
<dd>(1-UT2k4, 3-SOF2, 3-WET, 3 COD:UO, 1-CSS)<br>
<dd>Team Judge UT2004 67.18.122.226 7777 10000 24 8 ut4s xCTFGame
CTF-SMOTE
<dd>Team Judge Public CTF 67.18.122.226 20100 15000 30 0 sof2s ctf
mp_kam4
<dd>Team Judge Public Custom 67.18.122.226 20200 15000 30 0 sof2s ctf
mp_Italy3
<dd>Team Judge L337 Match Server 67.18.122.226 20300 10000 20 0 sof2s
ctf
<dd>arioche/mp_small
<dd>Team Judge 9 Map Campaign ETPro 3.0 67.18.122.227 27960 25000 28 1
ets et
<dd>supplydepot
<dd>Team Judge Custom/Stock 67.18.122.227 27961 100000 28 0 ets et
oasis
<dd>Team Judge Match Server 67.18.122.227 27962 15000 18 0 ets et
battery
<dd>Team Judge L337 COD/UO Server 67.18.122.228 28960 25000 24 5 cods
tdm
<dd>mp_saint-rush
<dd>Team Judge L337 COD/UO CTF-HQ 67.18.122.228 28961 25000 24 0 cods
ctf
<dd>mp_cassino
<dd>Team Judge L337 COD/UO Match 67.18.122.228 28962 25000 24 0 cods
dom
<dd>mp_arnhem
<dd>Team Judge L337 HL2 CS 67.18.122.229 27015 - 24 0 hl2s cs_italy<br>
<dd>You see the three UO servers are on ports 28960, 28961, and 28962.
Each
<dd>game is a separate install on Linux so the connection is defined
uniquely
<dd>for each.<br>
<dd>Dave<br>
<dd>At 01:46 PM 12/2/2004, you wrote:
<dd>> People are so
sensitive. Ever hear of filters? This is nothing
<dd>> compared to many many other list.
<dd>>
<dd>>
<dd>>Anyway, to bring this back to on-topic.
<dd>>
<dd>>I stumbled across one message here that mentioned running
multiple servers
<dd>>on one machine. I assume you must set different port
numbers for them.
<dd>>
<dd>>Can you run one as internet and the other as LAN?
<dd>>
<dd>>I would also assume that the bandwidth calculations applies to
both
<dd>>servers combined as well, correct?
<dd>>
<dd>>So if I had a Pentium III 550 running Slackware Linux and it has
100 BaseT
<dd>>connection to the world, my limitation would be the processing
power of
<dd>>that machine?
<dd>>
<dd>>It would also seem like CoD would do a lot of repetitive commands
(on the
<dd>>programming level) that a processor with a large cache would be
<dd>>best. Since AMD processors usually handle the repetitive
commands better
<dd>>than Intel, would it not make more sense to run an AMD
processor?
<dd>>
<dd>>Has anyone ran a Quake II server along side a CoD?
<dd>>
<dd>>Thanks
<dd>></blockquote>
<dd><tt>-------------------------------------------------------------------------------
<dd>Mark J.
DeFilippis
defilm@acm.org
<dd>
defilm@ieee.org<br>
</blockquote>
</dl>S1-------------------------------------------------------------------------------<br>
Mark J. DeFilippis, Ph. D
EE
defilm@acm.org<br>
defilm@ieee.org<br><br>
<br>
</body>
</html>