[p4] network appliance vs local disk?

[Richard Geiger]

Back in August of last year, there were some messages to this list on
the topic of whether it might make sense to store Perforce server data
on a Network Appliance filer, i.e., where some or all of the Perforce
server data is stored a NetApp filer, accessed by a Unix host running
a Perforce server.

Particularly alarming to me were statements to the effect that this
was obviously a bad thing to do, with dire Performance consequences,
and some anectdotal "we tried it one and it was awful" kinds of
sories.

Now, this was alarming to me mainly because I'm the Perforce
administator for a largish Perforce server (300 licenses, db.have
approaching 7Gb (might be time to recompact that one!) where we store
all server data - both the depot/ files as well as the db.* metadata
stuff - on a filer. My belief is certainly that it works well for us,
but I had no measurements to support that belief. (Also, I'll grant
that is might not work so well in certain configurations!)

In one message I wrote:
> Again, WRT performance vs. local disk, what we really need at this
> point is some real performance measurements. I hope to do some, and
> share my findings before long. I'll try to come up with some tools
> that will allow anybody to get some useful comparitive performance
> data for whatever configuration(s) that interest them.

Well, so much for "before long" :-), but I've finally found time to
make some progress on this front, having cobbled together a little
benchmarking framework that allows us to measure the performance of
certain Perforce operations in a controlled way, while varying
configuration parameters of particular interest. In this case, to see
the performance effects of server data on a locally host-attached
disk, vs. network attached (a NetApp filer).

The numbers below show preliminary results, for servers running on
both Compaq Alpha/OSF1 and Intel/Linux hosts.

There are 8 "runs" below;

The first 4 are with both p4d server and p4 client hosts on alpha/OSF1
hosts, two runs each where the storage (both host and client sides) is
on the host's local /usr filesystem, and two where the storage is on a
filer, access via NFS.

The second 4 runs are with the server on an Intel/Linux host (and the
client still being the same alpha/OSF1 host). Again, two runs each
with local disk storage, and two with network-attached.

For each "run", the Perforce server is initialized from scratch on the
"srv_host" machine; a client workspace is created on the "cl_host"
machine. A recent vintage gcc source distribution is then placed into
the client workspace, and all files in the distribution are "p4
add"ed.  The time to do this "p4 add" is recorded as "initial_add".
Next, a "p4 submit" is done (recorded as "submit"). Next, a second new
client workspace is created, and the gcc source tree is copied into it
("new_sync"). Finally, in an attempt to do a measurement that largely
factors out client-side performance, a pair of "p4 flushes" are done
from this second client ("flush_to_#none" and "flush_to_head").

In these tests, filer-based storage seems to compare favorably with
local disk. Of course, these comparisons are not the end-all for
measurements of this type - see the notes at the end; but in my mind,
they do seem to refute the idea that there's something wrong with the
concept of running a database of this sort to network-attached storage!

(The "L" lines are local disk, "N" are network-attached)

L 2001-02-09 08:14:27: config  : srv_host       castor
L 2001-02-09 08:14:27: config  : cli_host       saturn
L 2001-02-09 08:14:27: config  : p4bench_root   /usr/rmg/p4bench_root
L 2001-02-09 08:14:27: config  : p4bench_cli    /usr/rmg/p4bench_cli
L 2001-02-09 08:14:27: config  : cli_1_tar      /u/rmg/p4/p4bench/gcc-2.95.2.tar
L 2001-02-09 08:15:34: mark_tm :    55.69 seconds : initial_add
L 2001-02-09 08:19:35: mark_tm :   240.27 seconds : submit
L 2001-02-09 08:20:21: mark_tm :    45.83 seconds : new_sync
L 2001-02-09 08:20:40: mark_tm :     9.85 seconds : flush_to_#none
L 2001-02-09 08:20:48: mark_tm :     8.57 seconds : flush_to_head

N 2001-02-09 08:22:24: config  : srv_host       castor
N 2001-02-09 08:22:24: config  : cli_host       saturn
N 2001-02-09 08:22:24: config  : p4bench_root   /u/rmg/p4bench_root
N 2001-02-09 08:22:24: config  : p4bench_cli    /u/rmg/p4bench_cli
N 2001-02-09 08:22:24: config  : cli_1_tar      /u/rmg/p4/p4bench/gcc-2.95.2.tar
N 2001-02-09 08:23:11: mark_tm :    35.50 seconds : initial_add
N 2001-02-09 08:24:49: mark_tm :    97.58 seconds : submit
N 2001-02-09 08:25:13: mark_tm :    22.94 seconds : new_sync
N 2001-02-09 08:25:32: mark_tm :     9.29 seconds : flush_to_#none
N 2001-02-09 08:25:43: mark_tm :    10.84 seconds : flush_to_head

L 2001-02-09 08:29:10: config  : srv_host       castor
L 2001-02-09 08:29:10: config  : cli_host       saturn
L 2001-02-09 08:29:10: config  : p4bench_root   /usr/rmg/p4bench_root
L 2001-02-09 08:29:10: config  : p4bench_cli    /usr/rmg/p4bench_cli
L 2001-02-09 08:29:10: config  : cli_1_tar      /u/rmg/p4/p4bench/gcc-2.95.2.tar
L 2001-02-09 08:30:25: mark_tm :    63.22 seconds : initial_add
L 2001-02-09 08:34:26: mark_tm :   241.85 seconds : submit
L 2001-02-09 08:35:46: mark_tm :    78.60 seconds : new_sync
L 2001-02-09 08:36:03: mark_tm :     9.87 seconds : flush_to_#none
L 2001-02-09 08:36:13: mark_tm :     9.70 seconds : flush_to_head

N 2001-02-09 08:54:21: config  : srv_host       castor
N 2001-02-09 08:54:21: config  : cli_host       saturn
N 2001-02-09 08:54:21: config  : p4bench_root   /u/rmg/p4bench_root
N 2001-02-09 08:54:21: config  : p4bench_cli    /u/rmg/p4bench_cli
N 2001-02-09 08:54:21: config  : cli_1_tar      /u/rmg/p4/p4bench/gcc-2.95.2.tar
N 2001-02-09 08:55:11: mark_tm :    37.71 seconds : initial_add
N 2001-02-09 08:56:48: mark_tm :    97.82 seconds : submit
N 2001-02-09 08:57:13: mark_tm :    23.41 seconds : new_sync
N 2001-02-09 08:57:32: mark_tm :     9.23 seconds : flush_to_#none
N 2001-02-09 08:57:42: mark_tm :     9.76 seconds : flush_to_head

L 2001-02-09 09:14:58: config  : srv_host       bitter
L 2001-02-09 09:14:58: config  : cli_host       saturn
L 2001-02-09 09:14:58: config  : p4bench_root   /usr/rmg/p4bench_root
L 2001-02-09 09:14:58: config  : p4bench_cli    /usr/rmg/p4bench_cli
L 2001-02-09 09:14:58: config  : cli_1_tar      /u/rmg/p4/p4bench/gcc-2.95.2.tar
L 2001-02-09 09:16:12: mark_tm :    62.47 seconds : initial_add
L 2001-02-09 09:16:58: mark_tm :    46.65 seconds : submit
L 2001-02-09 09:17:42: mark_tm :    42.53 seconds : new_sync
L 2001-02-09 09:18:00: mark_tm :     8.89 seconds : flush_to_#none
L 2001-02-09 09:18:11: mark_tm :    10.95 seconds : flush_to_head

N 2001-02-09 09:19:54: config  : srv_host       bitter
N 2001-02-09 09:19:54: config  : cli_host       saturn
N 2001-02-09 09:19:54: config  : p4bench_root   /u/rmg/p4bench_root
N 2001-02-09 09:19:54: config  : p4bench_cli    /u/rmg/p4bench_cli
N 2001-02-09 09:19:54: config  : cli_1_tar      /u/rmg/p4/p4bench/gcc-2.95.2.tar
N 2001-02-09 09:20:45: mark_tm :    38.78 seconds : initial_add
N 2001-02-09 09:21:12: mark_tm :    27.08 seconds : submit
N 2001-02-09 09:21:38: mark_tm :    25.33 seconds : new_sync
N 2001-02-09 09:22:06: mark_tm :    12.45 seconds : flush_to_#none
N 2001-02-09 09:22:16: mark_tm :    10.37 seconds : flush_to_head

L 2001-02-09 09:29:56: config  : srv_host       bitter
L 2001-02-09 09:29:56: config  : cli_host       saturn
L 2001-02-09 09:29:56: config  : p4bench_root   /usr/rmg/p4bench_root
L 2001-02-09 09:29:56: config  : p4bench_cli    /usr/rmg/p4bench_cli
L 2001-02-09 09:29:56: config  : cli_1_tar      /u/rmg/p4/p4bench/gcc-2.95.2.tar
L 2001-02-09 09:31:10: mark_tm :    62.90 seconds : initial_add
L 2001-02-09 09:31:57: mark_tm :    46.30 seconds : submit
L 2001-02-09 09:32:40: mark_tm :    42.70 seconds : new_sync
L 2001-02-09 09:33:00: mark_tm :    11.77 seconds : flush_to_#none
L 2001-02-09 09:33:18: mark_tm :    17.20 seconds : flush_to_head

N 2001-02-09 09:39:37: config  : srv_host       bitter
N 2001-02-09 09:39:37: config  : cli_host       saturn
N 2001-02-09 09:39:37: config  : p4bench_root   /u/rmg/p4bench_root
N 2001-02-09 09:39:37: config  : p4bench_cli    /u/rmg/p4bench_cli
N 2001-02-09 09:39:37: config  : cli_1_tar      /u/rmg/p4/p4bench/gcc-2.95.2.tar
N 2001-02-09 09:40:23: mark_tm :    34.34 seconds : initial_add
N 2001-02-09 09:40:47: mark_tm :    23.57 seconds : submit
N 2001-02-09 09:41:14: mark_tm :    25.88 seconds : new_sync
N 2001-02-09 09:41:49: mark_tm :    12.86 seconds : flush_to_#none
N 2001-02-09 09:42:02: mark_tm :    13.39 seconds : flush_to_head


Notes:

  host "castor" is a 1 CPU 500Mhz alpha
    rmg $ rsh castor uname -a
    OSF1 castor.eng.netapp.com V4.0 1229 alpha

  host "saturn" is a 2 CPU 500Mhz alpha
    rmg $ rsh saturn uname -a
    OSF1 saturn.eng.netapp.com V4.0 1229 alpha

  host "bitter" is a 2 CPU 733Mhz Intel
    rmg $ rsh bitter uname -a
    Linux bitter 2.2.16 #2 SMP Tue Jun 27 16:48:08 PDT 2000 i686 unknown

  I believe the hosts are attached to the filer via a Gb-ethernet
  switch, and that p4 traffic between client and host is going
  via a switched 100baseT connection. (Need to verify this).

  The runs with p4benchh_srv and p4bench_cli "/usr/rmg" are
  the local-disk runs (whatever the machine's standard /usr
  filesystem happens to be on).

  The runs with p4benchh_srv and p4bench_cli "/u/rmg" are
  the net-attached runs. ("/u/rmg" is the filer where my
  home directory lives; it is, I beleieve, currently an F8xx
  series filer, and serves about 70 engineer's home directories.
  It was probably fairly lightly loaded when these tests were run.

  Before starting any run, the hosts involved were observed to be
  at a short-term load average of < .30; but some variation in
  load due to unreleated activity may have happened.

Of course, there are many factors that can affect performance that
aren't reflected in these tests; the Perforce databases used are very
small and young; the server is doing one thing at a time, and so
on. (I might extend the tests to vary these factors in the
future). But they are the best controlled comparisions I know of at
the moment as far as demonsrating performance effects of local
vs. network attached stroage with a Perforce server.

If anybody is interested in getting the scripts used to run these
tests (a bunch of perl), please let me know; I hope to put them on the
Perforce public depot at some point, but that happen happen sooned if
I know people are interested.

  - rmg