[ut2004] UT2004 benchmarking: what does the "rand" value mean?

Corey Hickey bugfood-ml at fatooh.org
Thu Apr 22 12:55:30 EDT 2004


Corey Hickey wrote:
> Xipher wrote:
> 
>>is their any correlation between the rand number, and the average FPS of
>>the benchmark?
>>
> 
> 
> Yes, actually, there is. The difference is slight, but definitely seems
> to be there. I was going to mention that, but wanted to get a larger
> sample size first.

Ok, I let the benchmark loop overnight and now I have 662 samples (I'm
not using the 50 values from my last report). I also stopped all
extraneous processes this time in the hope that there would be less
variation.

Here's the breakdown. The three avg values are the minumum, maximum, and
average values seen of the average fps value printed in the ut2004 logs.
"std. dev" is the standard deviation of the average fps values seen for
that particular rand value.


rand		freq	min avg.    max	avg.	avg. avg.	std. dev
1264169206	1	64.801750   64.801750 	64.801750	0
1425745073	11	65.049057   65.249557   65.165451	0.073945
1848445259	1	65.129051   65.129051	65.129051	0
1919451183	406	65.227013   65.669327	65.518138	0.079309
2051811764	61	65.155197   65.500549	65.375707	0.085930
231505642	1	65.030090   65.030090	65.030090	0
325730261	1	65.109085   65.109085	65.109085	0
460942436	1	65.039253   65.039253	65.039253	0
464066328	5	64.981461   65.140900	65.060403	0.062161
560406543	174	64.926521   65.258713	65.120059	0.075306

I'm suprised to see all those one-time rand values. Previously I had
only ever seen the four with the highest frequency. I don't know if
those values are anomalous, but I reported them for the sake of
completeness. I checked the log for each one of those to see if it
indicated a crash (the max fps is reported as inf when the benchmark
crashes) but each one looked good.

The value with a frequency of 5 also surprised me, because since that
rand value cropped up more than once, I'm sure it's not an anomaly.


Shutting down all the background processes gave me slightly higher
readings, but more importantly lowered the standard deviation to a
fraction of what it was before. I can now say with reasonable confidence
that different rand values correspond to different average fps values.
Take these two, for instance (copied from the above table).

rand		freq	avg. avg.	std. dev
1919451183	406	65.518138	0.079309
560406543	174	65.120059	0.075306

According to this page:
http://www.robertniles.com/stats/stdev.shtml
...two standard deviations from the mean acount for approximately 95% of
the samples. (I've never taken statistics so somebody please correct me
if I've messed something up) Multiplying the standard deviation by two
and then adding and subracting that result to and from the average gives
a 95th percentile range.

rand		freq	avg. avg.   std. dev	95th % range
1919451183	406	65.518138   0.079309	65.359520 - 65.676756
560406543	174	65.120059   0.075306	64.969447 - 65.270671

So, the 95% of samples closest to the mean for rand=1919451183 are all
higher than the 95% closest to the mean for rand=560406543. The
difference is slight, but definitely reliable.

If anyone else wants to look or check my work, here are all the logs:
http://fatooh.org/files/bigbench.tar.bz2


-Corey



More information about the ut2004 mailing list