[p4] Slow Downs - More RAM?

[David Ferguson]

Just wanted to add my 'second that' to Tony's comments.
We basically run into major delays in this area whenever DB size > 1/2
physical memory.  (exceptions exist of course, but pretty reliably...)
Problem appears to be mapped to 
P4 request A needs part of the DB that isn't in memory.
P4 A grabs a lock that everybody needs
System takes a long time to load in the appropriate DB files that A needs to
complete.
P4 request B,C,D,E,F,G,H,... All get c-switched into process table and play
'I can't go yet because lock is held' until it comes time for P4 request A
again, at which time system continues trying to load in the right DB files.
Eventually everything is there for A to finish, it does.  By which time there
is HIGHLY likely to be something else in the queued up transactions that
causes it all over again...

Check the wait time on various locks in the logfile to prove, or
Examine pagein/pageout traffic in synchronization with load average.  I've
decided any time that
Pgin is significantly higher (say 2X) than pgout
Indicates I've got the above problem.

FWIW -- Currently our DB is 130GB and RAM is 128GB

-daf
 

> -----Original Message-----
> From: perforce-user-bounces at perforce.com 
> [mailto:perforce-user-bounces at perforce.com] On Behalf Of Tony Sweeney
> Sent: Monday, January 08, 2007 12:51 AM
> To: Terry Metler
> Cc: perforce-user at perforce.com
> Subject: Re: [p4] Slow Downs - More RAM?
> 
> Terry Metler wrote:
> > I need opinions about hardware and how "big" I need to go...
> > 
> >  
> > 
> > We've been running Perforce now for about 7 months.  Up to November 
> > everything was running very smoothly but since then we have 
> many slow 
> > downs a day.  There are no indications on the system except that I 
> > have noticed just before everything goes back to normal 
> that the cores 
> > all peg to 100% utilization for a few seconds.  This has 
> slowly gotten 
> > worse to now it is happening two or three times a day.
> > 
> >  
> > 
> > After talking with Perforce support they indicated that it 
> looks like 
> > a case that the server is running out of resources.  There 
> are points 
> > when there are over 700 concurrent transactions.  Support mentioned 
> > that it looks like a large process is backing everything up and 
> > eventually it does finish and everything else runs really fast that 
> > was backed up behind that one process.
> > 
> >  
> > 
> > I've rebuilt the databases, defragged the drives, removed the audit 
> > file and moved the database files to a separate drive to 
> try and get 
> > some additional performance.  None of these have made a significant 
> > improvement.
> 
> First off, how large are your db.* files?  If they (or a 
> useful working
> set) will comfortably fit in memory, then that isn't the 
> issue, and more RAM won't help.
> 
> Next,
> 	have you tried setting server logging to level 2 or 3 
> and analysing the log using P4GLA, a graphical logfile 
> analyser that can be found in the public depot?  With this 
> you can identify the long-running processes that are holding 
> locks and causing everything else to stall waiting for the 
> locks to be released.  If you can identify the source of the 
> problem, then you can start trying to tune the server to 
> mitigate the problem.  In releases prior to 2006.1, 
> performing large integrations using either GUI pretty much 
> guaranteed this behaviour at one site I worked at.  The fix 
> for P4Win 2006.1 is described in the release notes as follows:
> 
> #82234
>      If the number of files in a changelist is greater than 
> 500 after an
>      integrate, P4Win simply refreshes the Depot and Pending 
> Changelist
>      panes rather than attempting to individually add all the files to
>      the changelist.  (Job @18168 - Bug)
> 
> An equivalent fix seems to have made it into P4V as well.
> 
> Tony.
> 
> > 
> >  
> > 
> > Currently...
> > 
> > Users: 160
> > 
> > Files: 460 000 (not including all revisions)
> > 
> > Server
> > 
> >             CPU: 2 x 3.0Ghz Xeon
> > 
> >             RAM: 4GB
> > 
> >             OS: Windows Server 2003 Standard
> > 
> >  
> > 
> > New server I'm looking at...
> > 
> > CPU: 2 x 2.6Ghz Opteron
> > 
> > RAM: 32GB (Can be upgraded to 64GB)
> > 
> > OS: Windows Server 2003 Enterprise
> > 
> >  
> > 
> > I am hoping to hear from some people that have been using 
> Perforce for 
> > a lot longer than we have.  I don't want to purchase a 
> bigger server 
> > and not get the performance I'm being told I would get with "a lot 
> > more" RAM (the current one is almost maxed out physically 
> and is maxed 
> > with the Standard OS).  All recommendations are more than 
> welcomed as 
> > I want to have a better feeling that it is the hardware that is 
> > causing the slow downs before shelling out for a new server.
> > 
> >  
> > 
> > Thanks in advance for your recommendations and feedback!!
> > 
> >  
> > 
> >  
> > 
> > Terry
> > 
> > 
> > 
> > The information transmitted is intended only for the person 
> or entity 
> > to which it is addressed and may contain confidential and/or 
> > privileged material. Any review, retransmission, dissemination or 
> > other use of, or taking of any action in reliance upon, this 
> > information by persons or entities other than the intended 
> recipient 
> > is prohibited. If you received this in error, please contact the 
> > sender and delete the material from any computer.
> > _______________________________________________
> > perforce-user mailing list  -  perforce-user at perforce.com 
> > http://maillist.perforce.com/mailman/listinfo/perforce-user
> > 
> > 
> > 
> _______________________________________________
> perforce-user mailing list  -  perforce-user at perforce.com 
> http://maillist.perforce.com/mailman/listinfo/perforce-user
>