CephFS as a replacement for NFS: Part 1

This is the first in a series of posts about CephFS. The overall goal is to evaluate and characterize the behavior of CephFS and determine if it can be a reliable replacement for NFS.

The current use case of NFS is 400G-1T 'stashes' shared from an NFS server to hundreds of Linux/Unix clients in an academic setting. In some cases these stashes are accessed by a single user on a single machine, in some cases dozens of users access them across dozens of machines.

Drawbacks to the current situation are the same as any situation involving NFS:

• Security is a joke
• Single über-powerful NFS filers present a SPOF
• Bigger and bigger filers get more and more expensive
• Forced to use proprietary and expensive ZFS on Solaris
• Backing up is becoming a problem as total dataset size becomes more than a tape backup system can really hold
• No tiering of storage. The whole dataset either goes on the fast disks or the slow disks

There are also some advantages of this system:

• NFS is old faithful
• Every operating system supports it, and usually pretty well
• NFS ipv6's like a champ
• It's already working
• Integrates well with pam, autofs, ldap
• Vendor, while expensive, is really good at fixing it
• ZFS allows 'thin provisioning' so that we can over subscribe. 
• ZFS allows full nfsv4 acls to be used (This could also go in the drawbacks section because extended acls cause much pain)

Some key advantages we hope to achieve with ceph:

• Clustering
• Replication of data at the ceph layer instead of RAID
• Authentication
• Tiering of disks/storage
• Setting different replication levels for different storage sets

The CephFS remote filesystem has capabilities roughly analogous to NFS. There is a single 'volume', it can be simultaneously mounted by multiple clients, it respects unix groups.

In the follow up posts to this one we will build out a test ceph cluster, build filesystems on it, mount them, and generally attempt to build feature parity with an NFS system.

Solaris redundant nfs mounts

Stardate 90840.8 Nfs servers with lots of disks, sharing files out to multiple clients, is a pattern followed everywhere. Eventually your users begin to not only use these remote files, but start to put nfs directories into their PATH variable. This causes problems whenever you need to patch or reboot the nfs server because all shells launched by users with PATHs that look into the nfs directories will hang forever(this does assume you are mounting hard).

You can minimize the effect of this by using redundant mount information.

Before:

/usr/local -ro,hard,intr,suid  bunny.cat.pdx.edu:/disk/forest/local

After:

/usr/local -ro,hard,intr,suid  bunny.cat.pdx.edu:/disk/forest/local,caerbannog.cat.pdx.edu:/volumes/cave/misc/usr-local

It is best to maintain an idea of a primary and secondary, at least for administration. Modify only the primary, and rsync to the secondary. Use readonly mounting to mount. This mount appears in ``mount`` like this:

/usr/local on caerbannog.cat.pdx.edu:/volumes/cave/misc/usr-local,bunny.cat.pdx.edu:/disk/forest/local remote/read only/setuid/devices/rstchown/hard/intr/xattr/dev=5a0000e on Wed Mar 27 15:35:21 2013

Note that this is mounted on both servers. Packets get sent to both servers and the first to respond with valid information is reported to the system. This can make for some bizarre weirdness if you use readwrite mounts.

It is totally possible to use something like drbd between nfs servers (not on Solaris, obviously), to make this doable with readwrite mounts. I have not done this personally.