[bf1942] Sucking up all memory in a loop.

[Fritz Elfert]

Well, first i never run game-servers as root, second i got a better 
method which works _reliably_ ;-) See the attached program. In your 
startup-script, put the following:

<your regular server-startup stuff - starting in background>
# Max Memory in MB
MMAX=768
BFPID=`ps -C bf1942_lnxded | awk '{print $1}'`
bfwatch $PFPID $MMAX

Since bfwatch uses /proc/<PID>/statm, it's Linux-specific. If needed
for BSD i think somebody can find a similar method to get a processes
memory usage. How it works:
It checks the memory-usage of the server every second and if the server 
exceeds it's memory-usage over the given parameter, it gets sent a SIGSEGV 
which in turn triggers the core-dump ...
I didn't test it right now cause i'm at work and have to program other
stuff but tonight i'll do some tests with it and then post backtraces 
here. Since the SIGSEGV gets sent externally, of course the offending
function is not necessarily at the top of the backtrace but at least it
should be on the stack. I'll do several tests and will try to find the
common parts of the backtraces ...

Something more tricky would be using a technique like valgrind and
hooking into the server's brk() calls on demand.
Ryan:
If you would need such a more elaborated instrumentation, let me know.

Ciao
 -Fritz

On Fri, 14 Feb 2003, SHiNTAR wrote:

> Hi have exactly the same problem. I posted a backtrace a while ago.
> What I did, is run the process as root, this way, it doesn't get killed 
> by the kernel.
> Your machine will be not-responding for a long time, but eventually it 
> will segfault and leave a core dump behind if ulimits are set to unlimited.
> 
> I think it would be nice for Ryan to have 2 of these issues backtraced 
> to confirm its really a problem
> 
> Fritz Elfert wrote:
> 
> >Hi,
> >
> >I tested 1.3 here with a 32-Player COOP setup which ran mostly fine with
> >latest 1.2. With 1.3 it never made it longer than the first map. Often it 
> >crashed before. I don't have a backtrace, because usually at some point it
> >starts sucking up all memory and if all virtual memory is used up, gets
> >killed by the kernel. Here is a snapshot of the last messages in the
> >console. At that time, the process used ~1.1G! of memory. Shortly after
> >that it got killed. (I have 1G RAM + 2G Swap). Needless to say that the
> >machine gets pretty unresponsive when this happens.
> >
> >STREAM_LOG: conn:0 S_GE:850 GEScoreMsg
> >STREAM_LOG: conn:0 S_GE:851 GEScoreMsg
> >STREAM_LOG: conn:0 S_GE:852 GEExitVehicle pId:244 ExitPlayer
> >STREAM_LOG: conn:0 S_GE:853 GESetDefaultVehicle pId:244 mVehicleId:490
> >STREAM_LOG: conn:0 S_GE:854 GENone
> >STREAM_DEBUG: exitvehicle USSoldier
> >STREAM_DEBUG: dead exit
> >STREAM_DEBUG: Bot dead: Roy Hunt
> >STREAM_DEBUG: entervehicle MultiPlayerFreeCamera
> >STREAM_DEBUG: Bot dead: Roy Hunt
> >FORMAT_DEBUG: setting hitpoints to -1
> >STREAM_DEBUG: Dead
> >STREAM_DEBUG: Remove
> >STREAM_DEBUG: Bot 17 registrered as dead @ 406.589 0
> >STREAM_LOG: conn:0 S_GE:855 GEScoreMsg
> >STREAM_LOG: conn:0 S_GE:856 GEScoreMsg
> >STREAM_LOG: conn:0 S_GE:857 GEExitVehicle pId:238 ExitPlayer
> >STREAM_LOG: conn:0 S_GE:858 GESetDefaultVehicle pId:238 mVehicleId:478
> >STREAM_DEBUG: Bot 17 dead 'Roy Hunt'
> >STREAM_DEBUG: No Old Target Found
> >
> >-Fritz
> >  
> >
> 
> 

-- 
Fritz Elfert <felfert at to.com>               Thinking Objects Software GmbH
Lilienthalstr. 2                                  Phone: +49 711 88770 400
70825 Stuttgart                                     FAX: +49 711 88770 449
--------------------------------------------------------------------------
-------------- next part --------------
#include <stdio.h>
#include <sys/types.h>
#include <signal.h>
#include <errno.h>
#include <stdlib.h>
#include <unistd.h>
#include <limits.h>

int main(int argc, char **argv)
{
    pid_t bfpid;
    unsigned long maxmem;
    FILE *stf;
    char buf[PATH_MAX];
    char fn[PATH_MAX];

    if (argc != 3) {
	fprintf(stderr, "Usage: bfwatch <BF1942PID> <MaxMBytes>\n");
	exit(-1);
    }
    sscanf(argv[1], "%d", &bfpid);
    sscanf(argv[2], "%lu", &maxmem);
    maxmem *= 1024; /* have MB, want KBytes */
    sprintf(fn, "/proc/%d/statm", bfpid);
    while ((stf = fopen(fn, "r"))) {
	unsigned long mem;

	fgets(buf, sizeof(buf), stf);
	sscanf(buf, "%lu", &mem);
	mem *= 4; /* have pages (a 4k), want KBytes */
	printf("Using: %d\n", mem);
	if (mem > maxmem) {
	    kill(bfpid, SIGSEGV);
	    printf("Sending SIGSEGV to process %d\n", bfpid);
	}
	fclose(stf);
	sleep(1);
    }
    return 0;
}