[p4] backups (long)

[Marc Lewert]

Our backup is probably not typical of a perforce installation, but I 
thought I would include it to give you a feel for the range of possible 
backup solutions.
Our goals included short lock times on a daily basis because we have 
development teams using the same server in multiple time zones.

We have two backups (one daily, one once a week).
The daily backup (normal peforce lock time less than a minute to 2 minutes):

    * We start with a journal-only save (p4d -jj), which takes us from
      under a minute to 2 minutes
    * We resync to server to a backup area in this order:
          o logs
          o licenses
          o depots
          o checkpoints and journals
    * Now we "proof" the backup to an off-line area
          o Load yesterday's checkpoint
          o Load all today's journal (or any journals since the last
            good checkpoint)
          o Create new checkpoint to the active checkpoint area (rsynced
            above)
    * Send email

Lock time for this is usual less than a minute, but can run to 2 minutes.
Total time for the run for us is in the range of 3.5 hours
Should the daily backup fail for any reason, we create a cookie so the 
next backup will be the full Once-a-week-backup.


The Once a Week Backup (also done the day following a daily backup failure):
Average lock time 25-30 minutes

    * Do journal-only save (p4d -jj)
    * Copy live database while DB is locked (p4d -c) to the off-line area
      This takes about 25-30 minutes
    * Rsync to the backup area in the following order:
          o logs
          o licenses
          o depots
          o checkpoints and journals
    * Validate the off-line database (currently p4d -xv, soon to include
      p4d -xx)
    * Create checkpoint to on-line area, from offline database (p4d -jc)
    * Send email

Lock time for perforce runs 25-30 minutes
Total time for us to do this procedure is 5.5 hours

Since we are in earthquake country, once the backup is complete, we 
rsync the entire backup area to a disaster recovery site.

We have a test perforce server which is identical in hardware to our 
primary server (it is also our backup hardware in case we need it), and 
we use it for testing procedures, triggers and scripts.  Whenever we do 
this, we load it with one of our backups. This verifies both that the 
backups are good and that the procedure to recover perforce from a 
backup is good.

We also irregularly bring up perforce at the disaster recovery site to 
make sure the data there is good and the procedure used to get it up works.


     marc


duncan.bowring at realtimeworlds.com wrote:
> Hey everyone,
>
> What do you folks do for backing up your p4 repository.. mainly curious
> to see what other people do out there.
>
> Hope to see some of you in London. ;)
>
>  
>
> Cheers
>
>  
>
>
> ____________________________________________________________________
> DISCLAIMER
>
> This message and any attachments contain privileged and confidential information intended for the use of the addressee named above. If you are not the intended recipient of this message, you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. Please note that we cannot guarantee that this message or any attachment is virus free or that it has not been intercepted and amended. The views of the author may not necessarily reflect those of Realtime Worlds Ltd.
>
>  
>
> Realtime Worlds Ltd is registered in Scotland, number 225628. Registered Office: 152 West Marketgait, Dundee, DD1 1NJ.
> _______________________________________________
> perforce-user mailing list  -  perforce-user at perforce.com
> http://maillist.perforce.com/mailman/listinfo/perforce-user
>
>   

-- 
Marc Lewert             Going Postal: TiVo Inc.
Senior Software Production Engineer   2160 Gold St.
Email: marc at tivo.com               San Jose, CA 95002
Phone: 408-519-9486                   Main Phone: 408-519-9100