UPDATE: [p4] Perforce running dog slow

Fri Sep 14 14:52:05 PDT 2001

Hi Perforce Users!!

Thanks for all who responded... the issue actually ended up to have nothing
to do with our server or network. It is a Perforce scalability issue.

To give you an idea of the size of our implementation, we have 100,000
files, 150,000 change lists, 375 users, approx 130,000 - 200,000 operations
a day. We have over 6,500 clients, 2,300 labels, 1,500 branches. At these
levels, certain operations do not scale. 

The biggest issues have been with "p4 client -d" and "p4 files". These
operations will cause all other processes to stack up behind them, shooting
the number of processes in the que way over average levels. Once the command
finally completes, 2-5 minutes later, the rest of the operations will begin
again. "p4 edit" of a small file that usually takes less than a second to
execute takes over 60-90 seconds to complete in this situation. The
solutions offered in the tech notes and by support, i.e. limiting
protections, making client sizes smaller, etc, do not scale either in a
large implementation, and have not improved these issues.

In the case of "p4 client -d", Perforce support explained it to me as a
locking issue. I guess Perforce was designed with the theory that deletion
of clients would happen infrequently. At Akamai, we tend to use and delete
clients often, an average of 200-600 per day. Because it was expected to be
run infrequently, it wasn't written to be as efficient as other operations,
i.e. p4 sync, which is expected to be run very frequently. Running p4 client
-d flushes the db.have table of all file listings for that client. It then
deletes the client. It is inefficient because the operation locks the entire
database, and no other operations can occur during this time. If there is a
large amount of files to delete, this operation may take a long time.  

To alleviate this, we have implemented some workarounds in our automated
processes. For the "p4 client -d" problem, we have done "p4 flush
//...#none" or "p4 sync //...#none". Even these operations slow things down,
so I am running the commands for each depot, and in many cases, each
directory under the larger depots. i.e. "foreach dir in `p4 dirs //depot/*`
do p4 flush $dir/..." Yuck!!  For the "p4 files" problem, there is not much
I can do, except monitor the server and educate folks that these commands
clog things up. In automated scripts, we have alleviated much of the problem
by running p4 files on a depot path basis, i.e. "p4 files
//depot/main/... at label" rather than "p4 files @label". But, something like,
"p4 files //...*somestring*... hangs the server for minutes at a time...
Yuck!!

I'd be happy to share more of how we diagnosed this if anyone is
interested... Thanks again for your help!! Has anyone else had similar
scaling issues?

Nicole

> -----Original Message-----
> From: Wong, Leonard 
> Sent: Wednesday, August 08, 2001 11:46 AM
> To: Sweeney, Nicole
> Subject: RE: [unix-admins] FW: [p4] Perforce running dog slow
> 
> 
> FYI-
> 
> Most of the corporate network in Santa Clara is port speeds 
> are set to Full/100.
> 
> If you want to test changing the port speed back to 
> auto/auto, let me know and I can make those changes in Santa 
> Clara (or in San Mateo)
> 
> -----Original Message-----
> From: Sweeney, Nicole [mailto:nicole.sweeney at akamai.com]
> Sent: Wednesday, August 08, 2001 10:46 AM
> To: P4admin (E-mail); unix-admins, Dist. List; DeLucia, Dante;
> Stodolsky, Daniel
> Subject: [unix-admins] FW: [p4] Perforce running dog slow
> 
> 
> 
> 
> > -----Original Message-----
> > From: Todd Short [mailto:tshort at cisco.com]
> > Sent: Wednesday, August 08, 2001 6:25 AM
> > To: nicole.sweeney at akamai.com; perforce-user at perforce.com
> > Subject: RE: [p4] Perforce running dog slow
> > 
> > 
> > We had a similar problem... we discovered it was our network 
> > switches that
> > were causing the slowdown. After our acquisistion, the port 
> > speeds on our
> > switches were locked down to a certain speed, and it wasn't 
> > auto-config
> > anymore. Even locking the ports to 100 Mbps didn't help, we 
> > had to configure
> > it to auto to get our speed back. What used to take 10 
> > minutes now took 18
> > seconds.
> > 
> > Have any network changes been made recently?
> > 
> > --
> > -Todd Short
> > // tshort at cisco.com
> > // "One if by land, two if by sea, three if by the Internet."
> > 
> > 
> > > -----Original Message-----
> > > From: perforce-user-admin at perforce.com
> > > [mailto:perforce-user-admin at perforce.com]On Behalf Of 
> > Sweeney, Nicole
> > > Sent: Tuesday, August 07, 2001 10:06 PM
> > > To: 'perforce-user at perforce.com'
> > > Subject: [p4] Perforce running dog slow
> > > 
> > > 
> > > In recent weeks, perforce has begun to run very slow on our 
> > > site. Operations
> > > that should take seconds are taking minutes. 
> > > 
> > > We are running Solaris 2.7 with a NetApp, 2gigs of RAM. The 
> > > CPU stays very
> > > low on each process, usually maxing out at 5% at most. We 
> > have 100,000
> > > files. Our db.rev table is 559Megs.
> > > 
> > > Based on the admin guide and the tech notes, we have enough 
> > > memory. p4 info
> > > response is usually very prompt. At this point, we have been 
> > > grasping at
> > > straws, but here's what we've seen:
> > > 
> > > -- sync, submit, SubmitChange, CommitSubmit, client -d 
> > > operation take a very
> > > long time and processes, sometimes upwards of 100 stack up 
> > > behind it in the
> > > que
> > > -- CPU loads stay very low
> > > -- our protections are set pretty tight
> > > -- NFS traffic averages 30% of 100Mb connection btwn the 
> > > netapp and the
> > > server
> > > -- swap is varying btwn 1503000 and 64000 free.
> > > -- reboots seem to help but we haven't confirmed
> > > 
> > > 
> > > Help??
> > > 
> > > Nicole Sweeney
> > > Software Configuration Engineer
> > > Akamai Technologies, Inc.
> > > (858) 909-3654 direct
> > > (858)442-7011 cell
> > > 
> > > _______________________________________________
> > > perforce-user mailing list  -  perforce-user at perforce.com
> > > http://maillist.perforce.com/mailman/listinfo/perforce-user
> > > 
> > 
> 