[p4] What actually happens during checkpoint

Brad Holt brad.holt at autodesk.com
Thu Mar 5 07:58:03 PST 2009


This should be an interesting thread.  I'm no expert, but I'm guessing that  "... the checkpoint would write the transactions in the same order that they were originally done..." would be off, if by that you mean it uses the transaction log (the journals).  The checkpoint doesn't need/use the transaction journals to create itself.  Similarly, the act of taking a checkpoint is the handiest way to check the veracity of the database, so it shows that it is mining the metadata directly.  This also means that you're right that "...a checkpoint is also not just a compressed copy of all the tables..." as that wouldn't check the db integrity.

As a benchmark, one of my servers is about your size - 150GB metadata, 5.75GB compressed checkpoint takes about 2.5 hours on Windows2003x64, metadata volume NTFS RAID 1/0 on its own channel.  I wouldn't reckon the RAM is the problem.  When I checkpoint, my memory usage doesn't go anywhere much (but it's well handy for the day to day grind).  If you've got the room, maybe try taking an uncompressed checkpoint just to make sure compression hasn't become the bottleneck.

Sorry for the non-answer, but if you get one offline or from p4, be sure to share it.  I'd be the better for it.

-----Original Message-----
From: perforce-user-bounces at perforce.com [mailto:perforce-user-bounces at perforce.com] On Behalf Of Jamie.Echlin at barclayscapital.com
Sent: Thursday, March 05, 2009 7:23 AM
To: perforce-user at perforce.com
Subject: Re: [p4] What actually happens during checkpoint

I should have mentioned we're using ext3, and it appears mounting it
with data=writeback would help, and XFS help even more. But even so, I'm
currently interested in why this process seems to have degraded so much
over the last 6 months (checkpoint used to take about an hour), and for
personal curiousity I'm interested in the mechanics of a checkpoint.

Cheers, jamie 

> -----Original Message-----
> From: perforce-user-bounces at perforce.com 
> [mailto:perforce-user-bounces at perforce.com] On Behalf Of 
> Echlin, Jamie: IT (LDN)
> Sent: 05 March 2009 14:26
> To: perforce-user at perforce.com
> Subject: [p4] What actually happens during checkpoint
> 
> Can anyone help me understand what happens during a checkpoint?
> Background is, the checkpoint procedure has got incredibly 
> slow, about 9 hours to produce a 4.5 G compressed checkpoint 
> from a 104G database.
> 
> We have some problems with our SAN. The LUNS that make up the 
> data volume are split over two different array controllers, 
> one of them in a different data centre. However, this 
> shouldn't cause problems in itself, it's just not "best 
> practice", or so I've been told.
> 
> My theory about the performance then was that the checkpoint 
> would write the transactions in the same order that they were 
> originally done, causing reads over all the different tables, 
> somehow exacerbating a disk problem. Also we don't have 
> enough main memory, only 32G. However, reading more, 
> checkpoints are not the same as journals, the transactions 
> are not in the same order. But a checkpoint is also not just 
> a compressed copy of all the tables, if that was true there 
> was no way it could so long. So what gives? ;-)
> 
> A restore of a checkpoint takes 10 hours, so not much 
> difference between reads and writes. Just to rule out 
> decompressing being the bottleneck, it's only using a couple 
> per cent of CPU.
> 
> Cheers, jamie
> 
> 
> _______________________________________________
> 
> This e-mail may contain information that is confidential, 
> privileged or otherwise protected from disclosure. If you are 
> not an intended recipient of this e-mail, do not duplicate or 
> redistribute it by any means. Please delete it and any 
> attachments and notify the sender that you have received it 
> in error. Unless specifically indicated, this e-mail is not 
> an offer to buy or sell or a solicitation to buy or sell any 
> securities, investment products or other financial product or 
> service, an official confirmation of any transaction, or an 
> official statement of Barclays. Any views or opinions 
> presented are solely those of the author and do not 
> necessarily represent those of Barclays. This e-mail is 
> subject to terms available at the following link: 
> www.barcap.com/emaildisclaimer. By messaging with Barclays 
> you consent to the foregoing.  Barclays Capital is the 
> investment banking division of Barclays Bank PLC, a company 
> registered in England (number 1026167) with its registered offi!
>  ce at 1 Churchill Place, London, E14 5HP.  This email may 
> relate to or be sent from other members of the Barclays Group.
> _______________________________________________
> 
> _______________________________________________
> perforce-user mailing list  -  perforce-user at perforce.com 
> http://maillist.perforce.com/mailman/listinfo/perforce-user
> 
_______________________________________________

This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered offi!
 ce at 1 Churchill Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________

_______________________________________________
perforce-user mailing list  -  perforce-user at perforce.com
http://maillist.perforce.com/mailman/listinfo/perforce-user




More information about the perforce-user mailing list