[p4] Backup scripts?

[Weintraub, David]

It isn't unusual for the system software to be unable to do a simple
backup due to database consistency issues. You cannot backup databases
without locking them first. When I was using ClearCase, you had to lock
the database, and backup everything in one shot. The lock could last
hours, and maybe even all night. I've worked with very large Oracle
installations where the backup could take all day. The only way to do a
backup was to take the database down for the entire day.

In larger setups, you do have mechanisms for doing backups without
taking down user access to the database for days at a time. NetApps, for
example, takes a snapshot of your disk at a particular moment in time.
You then backup from the snapshot. Other systems use disk mirroring
where one mirror will be taken down, a backup done, then the mirror
would be brought back up and resync'd with the other disk. We used this
method to backup our Oracle database (we had three mirror drives, and
would only do the backup once all three mirrors were in sync.)

Of course, most places I've worked at haven't had such a large setup.
Unfortunately, most tempted fate by backing up without locking the
database, and a few paid the price.

One shop was a ClearCase shop that insisted they couldn't afford to have
ClearCase locked while the backup took place. I insisted that if they
couldn't afford to lock down their ClearCase database for the seven
hours it would take to do the backup, then we needed to put the whole
thing on a NetApp, do a checkpoint, and backup the checkpoint. Nope,
they couldn't afford a NetApp either.

One day, we had a disk problem and were forced to restore from the
backups. The backups proved no good, and the whole shop was shutdown for
almost a week (100+ developers sitting idle) while I and three other
SCMs slowly pieced together our ClearCase installation. Let's say that
each developer costs you $70 per hour. That's 100 developers at 40 hours
of thumb twiddling, and you multiply that out and... What do you know!
They realized they could afford a NetApps after all. Or, maybe a better
way to put it, they couldn't afford not to have one any more.

At least Perforce provides a good mechanism to allow for live backups.
You can do a checkpoint, and then a backup without ever taking down the
server. If you don't like doing a checkpoint, then you need to take down
the p4d service and backup the database, the checkpoint, the journal,
and the license with the p4d service down. What you should never do is
tempt fate by backing up an inconsistent database. Otherwise, as one
former employer discovered, you're just wasting your time.

-----Original Message-----
From: Arnt Gulbrandsen [mailto:arnt at gulbrandsen.priv.no] 
Sent: Monday, September 25, 2006 9:21 AM
To: perforce-user at perforce.com
Cc: Weintraub, David
Subject: Re: [p4] Backup scripts?

I should have known this would happen.

My preferences are:

1. If the server's OS/architecture may change between backup and
restore, use p4 checkpoints.

2. Else, if the site is so small that one can trust the server to be
idle at night, just rsync and be done. (I draw the limit at, ah, 1.5
developers, or 2.5 if it's acceptable that a backup fails once per
year.) This would seem to apply to the original poster.

3. Else, if it's bigger and the p4d runs on a real OS, mount the
relevant file system somewhere as a snapshot and back up the snapshot.

4. If all else fails, use p4 checkpoints.

So why don't I like p4 checkpoints? I've never thought about it very
much. I think the basic reason is that IMO it's wrong for userspace code
like p4d to take care of this. It's a perfectly ordinary backup. 
Not something each application should have to handle. I also don't like
backing up something other than what's used.

Arnt
(whose p4ds run on a real OS)