[p4] NFS mounted depot

[Weintraub, David]

I talked to Perforce one more time. I explained about the debate on the
mailing list which they knew about, and I wanted an explanation of what
Perforce actually supports, and if they do support NFS, why the warning
in the Admin manual. Here's the lowdown:

* Perforce concedes that the warning in the Administrator's manual could
be out of date. As was pointed out to me, testing was done on Solaris 6
which was released quite a while ago. Not all parts of the manual are
updated with each release. It is likely that warning on NFS on Open
Source systems is no longer a problem.

* I have a friend who is a senior System Administrator confirm that the
warnings about Linux NFS locking are indeed out of date. In the latest
Redhat distributions, Linux is usually be configured with strong NFS
file locking. It all depends upon your release of NFS. The weak file
locking warning is true for NFS 2 which had no file locking mechanism.
However,  NFS 3 and NFS 4 do file locking. (See
http://nfs.sourceforge.net/ for details). My friend was unsure whether
this also applies to BSD distributions. However, it is easy enough to
check your version of NFS and to see if the nfs locking daemon is
running.

* I believe that NetApps and other types of similar disk arrays storage
areas do their own locking beyond what NFS does. This is why NetApps was
not overly concerned with this issue when they implemented Perforce on
their own disk storage system.

* Perforce's actual recommendation is that all files should be stored
locally for best performance and security. In other words, it really
doesn't matter what is your operating system or the version of NFS you
are using, Perforce recommends that you simply don't use NFS.

* The *.db files are especially vulnerable to damage due to problems
with NFS. Perforce accesses records in the middle of the files, deletes
records in the middle of the file, and moves records around these files.
Two processes doing this at the same time can be a disaster.

* The Journal file mainly has performance issues on NFS mounts. A single
transaction might cause 50 or more journal entries with each one needing
NFS access. Small delays due to NFS add up quickly and ruin performance.

* Perforce recommends that Journal file and the *.db files be kept on
two different disks using different controllers. By keeping the Journal
and *.db files on separate controllers, you minimize the possible
problems with a system crashing and irrevocably losing information. If
the *.db files are damaged, you can rebuild them with the journal. If
the journal file is damaged, it will be rebuilt with the *.db files.
Perforce has said they've seen disk controllers fail and damage all
disks on that controller, but multiple disk controller failures are
much, much rarer.

* The RCS files are the least vulnerable to non-locking NFS issues since
these files are read and written as whole units. Plus, the likelihood of
different server processes having collisions are a lot less.

* If you *have* to use NFS disks for any part of your depot, it is
recommended that journal and *.db files be kept locally, and you put the
RCS files on NFS. This causes the least performance problems, and is the
best way to handle issues with non-locking versions of NFS.

* Finally, despite all of these warnings, posturing, recommendations,
and hemming and hawings. Perforce did want to leave me with one message:
They've never walked away from a customer because of the way they've
implemented Perforce. Whenever there is a problem, they are there to fix
it.

I guess it really all comes to tradeoffs. I would love a high-speed,
multiprocessor, dedicated Perforce server with high-speed, local,
dedicated disk access with unlimited storage. Then again, I also want a
million dollars and a pony. It is highly unlikely I'll get the money, my
pony, or my dream Perforce server, so I'll take what I can.

NFS storage for the RCS files is reasonable for me. The NFS disks are
automatically backed up and a snapshot is taken every hour and stored
for 2 days. This makes the tradeoff in performance and other issues
worth it for me. However,  I can make a good argument to our IT
department about keeping our *.db files and journal files locally. They
don't have to back up the journal or the *.db files, and storing these
files locally vs. remotely greatly improves the performance. If I can
actually convince IT to store these on two different disk controllers,
I'd be in heaven. I'll see if I can at least store these on two separate
physical disks.