[p4] Perforce server lockup problem

Kenneth Olwing kolwing at bea.com
Mon Mar 14 01:15:35 PST 2005


> Our server runs on linux and nothing else but perforce runs on it. We
used to have a 1+0 SATA raid (via motherboard)
> however it did not prove stable (drives would disconnect) so now we
use straight SATA.

A short story on a Linux lockup:

About a year ago we tried upgrading our trusty old Linux to RH 9
Enterprise. It mostly worked, but also exhibited freezes (the 'press the
powerbutton' kind). After a day I simply called it a failure and went
back to the old server pending investigation.

It ended up being a memory overcommit situation that killed Linux.
What's worse, I could prove it was something that could be done by an
arbitrary user (i.e. for example the account running p4d). By just
logging in like John Doe I could execute:

perl -e 'while(1){push(@test, qq(test) x 1024)}'

I.e. basically just keep allocating memory. Look at the machine with top
and you'll see it quickly eat up your memory, then swap space
and...eventually it became a vegetable. Ok, hooking up a terminal to the
serial port leaves that option and you could kill the offending process,
but lacking that you'd be hosed; power button the only recourse :-(

Support gave us a workaround...do (as root):

echo 2 > /proc/sys/vm/overcommit_memory

Now, changed working style nicely kills the process if it runs away.

It's most likely fixed now (but I'm honestly not sure, interest was
lukewarm in fixing it...and that was with a paid version of RH and it
was from a just basic install of the OS...scary), but you might want to
check for this being a factor as p4d sometimes gobbles up large amounts
of memory...

ken1




More information about the perforce-user mailing list