[ut2004] UT2004 benchmarking: what does the "rand" value mean?
Corey Hickey
bugfood-ml at fatooh.org
Thu Apr 22 12:55:30 EDT 2004
Corey Hickey wrote:
> Xipher wrote:
>
>>is their any correlation between the rand number, and the average FPS of
>>the benchmark?
>>
>
>
> Yes, actually, there is. The difference is slight, but definitely seems
> to be there. I was going to mention that, but wanted to get a larger
> sample size first.
Ok, I let the benchmark loop overnight and now I have 662 samples (I'm
not using the 50 values from my last report). I also stopped all
extraneous processes this time in the hope that there would be less
variation.
Here's the breakdown. The three avg values are the minumum, maximum, and
average values seen of the average fps value printed in the ut2004 logs.
"std. dev" is the standard deviation of the average fps values seen for
that particular rand value.
rand freq min avg. max avg. avg. avg. std. dev
1264169206 1 64.801750 64.801750 64.801750 0
1425745073 11 65.049057 65.249557 65.165451 0.073945
1848445259 1 65.129051 65.129051 65.129051 0
1919451183 406 65.227013 65.669327 65.518138 0.079309
2051811764 61 65.155197 65.500549 65.375707 0.085930
231505642 1 65.030090 65.030090 65.030090 0
325730261 1 65.109085 65.109085 65.109085 0
460942436 1 65.039253 65.039253 65.039253 0
464066328 5 64.981461 65.140900 65.060403 0.062161
560406543 174 64.926521 65.258713 65.120059 0.075306
I'm suprised to see all those one-time rand values. Previously I had
only ever seen the four with the highest frequency. I don't know if
those values are anomalous, but I reported them for the sake of
completeness. I checked the log for each one of those to see if it
indicated a crash (the max fps is reported as inf when the benchmark
crashes) but each one looked good.
The value with a frequency of 5 also surprised me, because since that
rand value cropped up more than once, I'm sure it's not an anomaly.
Shutting down all the background processes gave me slightly higher
readings, but more importantly lowered the standard deviation to a
fraction of what it was before. I can now say with reasonable confidence
that different rand values correspond to different average fps values.
Take these two, for instance (copied from the above table).
rand freq avg. avg. std. dev
1919451183 406 65.518138 0.079309
560406543 174 65.120059 0.075306
According to this page:
http://www.robertniles.com/stats/stdev.shtml
...two standard deviations from the mean acount for approximately 95% of
the samples. (I've never taken statistics so somebody please correct me
if I've messed something up) Multiplying the standard deviation by two
and then adding and subracting that result to and from the average gives
a 95th percentile range.
rand freq avg. avg. std. dev 95th % range
1919451183 406 65.518138 0.079309 65.359520 - 65.676756
560406543 174 65.120059 0.075306 64.969447 - 65.270671
So, the 95% of samples closest to the mean for rand=1919451183 are all
higher than the 95% closest to the mean for rand=560406543. The
difference is slight, but definitely reliable.
If anyone else wants to look or check my work, here are all the logs:
http://fatooh.org/files/bigbench.tar.bz2
-Corey
More information about the ut2004
mailing list