[p4] Disaster Recovery

Weintraub, David david.weintraub at bofasecurities.com
Tue Jul 25 06:27:11 PDT 2006


There are two parts to Perforce: 1). The depot of RCS source files that
store the actual source and changes, and 2). The database. Both are
important to backup. Unfortunately, because the way the database is
setup, there is no guarantee that restoring all the database files as is
will work (especially if the machines use different architectures and
OSs).

Fortunately, Perforce allows you to "checkpoint" the database.
Checkpointing the database makes a text readable copy of the records in
the database that can be used to recreate the database. It is
architecture independent. Another part of the database backup strategy
is the Journal that is created based upon the last checkpoint. This is
also a text readable copy of the transactions since the last checkpoint,
and is also architecture independent.

At any time, you can recreate the entire database by using the last
checkpoint and the current journal. Of course, if your disk gets fried,
and you have the last checkpoint and current journal on your current
disk, you will lose them as well.

Therefore, it is important to make "checkpoints" of your database on a
regular basis, and backup the checkpoints as well as the source data.
This can be done while Perforce is up and running although it might slow
down Perforce, so it is best to do this when there is minimum activity.
It takes about 5 minutes to do our checkpoint, but then our Perforce
archive is only a few months old. I expect this time will grow
substantially as we get more transactions in our Perforce archive.

Since we checkpoint in the middle of the night, I don't know all the
repercussions of checkpointing. I believe you are allowed to sync from
the archive, do integrations, etc. However, I am not sure whether you
can "submit" changes during a checkpoint. You might be able to, but the
submit won't start until the checkpoint is complete. Someone with a bit
more experience can help me out there.

Anyway, when you make a backup of Perforce, you do a checkpoint, backup
the checkpoint, and then backup the archive. This all can be done live
since the database and source RCS files don't have to be in sync. The
source files can be ahead of the checkpoint in time. If you restore from
the last checkpoint, and there are extra "transactions" in the RCS depot
files, those transactions will be ignored, and your source archive will
be in the same state as the last checkpoint. If you really, really, need
a particular version of a file that's not in the checkpointed database,
it is easy enough to extract it from the RCS file, and submit it back
into Perforce.

Let me describe our setup. It is not the way Perforce recommends because
the database and depot are stored on NetApps boxes and not local disks.
However, there are a few white papers that describe a similar setup at
http://perforce.com.

* Hardware: We have two Linux machines running NFS Version 3 (which
supports file locking) and Kernel version 2.4. Both machines are
identical, and both are 100% dedicated as Perforce servers.

* The Perforce environment (including binaries for Perforce and trigger
scripts) are stored on a automounted directory which is on a NetApps
box. Also stored on this box is the RCS source depot, the journal, and
the database. These all are snapshotted every hour, and the snapshots
are stored for 48 hours. These servers are backed up every night, and a
copy of the backups are stored off site.

* I do a nightly checkpoint (and verification) of our Perforce database,
and this checkpoint is stored on the same NetApps box. It is also
snapshotted every hour, and backed up nightly.

* Stored in the same directory as the database are two Perforce license
files. One is for the primary server (called license-scsmerldcs01) and
the other for the secondary server (license-scsmerldcs02). We have a
symbolic link from the primary license to a link called "license".

* The two boxes have DNS names (scsmerldcs01 and scsmerldcs02). However,
we also have a DNS alias (merlin-perforce) pointing to the primary
server. Our users set P4PORT to the DNS alias and not to the actual DNS
name.

In case our primary server goes down, I can quickly change the symbolic
license link to point to our backup server license, change the DNS alias
from our primary server to our backup server, and get Perforce up and
running again with in an hour. The users will not have to change their
P4PORT variable. Once the primary server comes back on line, we can
change the DNS and license to point back to our primary server.

In the very unlikely event that the NetApps goes down (Since 1995 when I
worked with my first NetApps, I have *NEVER* seen a NetApps box go
down), we can restore to a new NetApps from the backup and recreate the
database from the last checkpoint. We will lose all the changes since
the last checkpoint. However, because our users will still have their
Perforce client root directories, they should have most of the changes
that were in the archive. Putting these versions back in is not too much
of an issue.

I use the fact that we have two Perforce servers and a backup license to
run my testing. I will copy the most recent snapshot of our Perforce
database and RCS archive to another directory, change the license in the
backup directory to point to the backup server, and startup the copied
Perforce instance on our backup server. I have a complete duplicate of
our Perforce instance that is no more than an hour out of date. I can
test triggers, data recovery, branching and merging strategies, etc. It
is a great test bed.

As far as Perforce proxies are concerned: They contain no useful
information to backup. They use the same database as the main Perforce
server, and only cache copies of commonly used files and versions. If
you delete all the files in a Perforce proxy cache, it will
automatically rebuild itself. In fact, you MUST clean out the cache
every once in a while or else you will run out of disk space. Once a
version is in a P4P cache, it is never deleted.

-----Original Message-----
From: perforce-user-bounces at perforce.com
[mailto:perforce-user-bounces at perforce.com] On Behalf Of Heizler, Mark
Sent: Tuesday, July 25, 2006 7:31 AM
To: perforce-user at perforce.com
Subject: [p4] Disaster Recovery

We are new to Perforce and are trying to figure out what the best way to
set up perforce for disaster recovery.  Backing up and restoring the
depot is always an option but I have been reading about Perforce Proxy
and wonder if anyone is using that or maybe some form of replication.

 

Thanks in advance,

 

Mark

_______________________________________________
perforce-user mailing list  -  perforce-user at perforce.com
http://maillist.perforce.com/mailman/listinfo/perforce-user



More information about the perforce-user mailing list