SLASH2 filesystem

Slash2 is a distributed filesystem where I/O servers and metadata servers can be at multiple sites. Files can be replicated or migrated between the I/O servers. Our deployment of slash2 is called /supercell and that is how it is mounted on all clients.

Full documentation: https://github.com/pscedu/slash2/wiki/

Binaries

The slash2 binaries have been installed at /opt/sam/slash2/build. /opt/sam should be mounted from s-misc0.frank.sam.pitt.edu onto any machine that is connected to the the /supercell deployment.

/opt/sam/slash2/PSC is a clone of the pfl repository.

The following scripts should also be added to /etc/profile.d/ on every machine mounting or serving /supercell

slash2.csh

slash2.sh

The scripts are also stored at /opt/sam/slash2/scripts

Deployment

The full deployment repository has been cloned to /opt/sam/slash2/PSC/inf/pgrrsc. These files contain information about I/O, MDS and client servers. All changes to these files must be made in /opt/sam/slash2/PSC/inf/pgrrsc and committed. The changes are deployed by running make in /opt/sam/slash2/PSC.

After the changes are deployed the pgrrsc.slcfg file, which is available from the deployment repository, will be installed at /opt/pittsam.s2/slcfg. This file stores information about MDS and I/O servers. Any changes made to this file require a complete outage of all I/O, MDS and client servers.

The client hostnames are stored in pgrrsc.dcfg on the MDS and IO servers. This information is installed to /opt/pfl_daemon.cfg on the MDS and I/O servers.

Client machines only need access to binaries and executables. No deployments files are installed.

Health

on supercell-mds0

slmctl -sc

O means online

Updates

TBD

Servers

At Pitt there are two I/O servers and one MDS. The configuration files necessary for the /supercell implementation are also on github

Name Role disk size hostnames NOTES
supercell-mds0 MDS N/A supercell-mds0.frank.sam.pitt.edu
supercell-mds0-priv.frank.sam.pitt.edu
does not mount /opt/sam
binaries at /opt/psc
ice IO 100GB supercell0.frank.sam.pitt.edu
supercell0-priv.frank.sam.pitt.edu
mounts /opt/sam
pan IO 43GB supercell1.frank.sam.pitt.edu
supercell1-priv.frank.sam.pitt.edu
does not mount /opt/sam
binaries at /opt/psc

Clients

The following machines are configured as clients. Configuration is stored in the deployment configuration repository

Public Hostname Private Hostname
tcga.sam.pitt.edu tcga-priv.sam.pitt.edu
virtuoso.sam.pitt.edu virtuoso-priv.sam.pitt.edu
login0a.frank.sam.pitt.edu login0a-priv.frank.sam.pitt.edu
login0b.frank.sam.pitt.edu login0b-priv.frank.sam.pitt.edu
app1.sam.pitt.edu --
clcbio-stage.sam.pitt.edu --

Notes from Kim Wong

  1. build as root from login0a
  2. cd /opt/sam/slash2/src
  3. make up or git pull
  4. make build; make install
  5. pkill -f sliod; /opt/sam/slash2/build/sbin/sliod.sh on supercell0.frank.sam.pitt.edu and supercell1.frank.sam.pitt.edu.
  6. on head0a, issue for i inseq 0 63; do bpsh n$i pkill mount_slash; done
  7. The other clients are: login0a, login0b, tcga-priv.sam, tcga-priv-new.sam, and the Pitt DTN
  8. For the MDS, pkill slashd
  9. DTN: rd-dtn0-prod-sam.cssd.pitt.edu
  10. /opt/sam/slash2/src/inf/pgrrsc/pgrrsc.dcfg

Notes from Fangping Mu

  1. Because of library configuration problem, I installed a fuse library to /opt/sam/sys/lib. If fuse library is required to be updated, reinstallation of new version of fuse library to this folder is required.
  2. Check IO server usage: on supercell-mds0, run slmctl -ss | head, W = "write-disabled"
  3. reject writes in a sliod near full: on mds0, slmctl -p sys.resources.PITT.ice.disable_lease=1 and slmctl -p sys.resources.PITT.pan.disable_lease=1
  4. setup selftest:
logon supercell0, `fshealthtest /data/supercell0` and `echo $?`, 
 
[root@supercell0 /]# fshealthtest -u 90 /data/supercell0 
[root@supercell0 /]# echo $? 
1 
 
modify /opt/sam/slash2/src/inf/pgrrsc/pgrrsc.slcfg
 
and change 
+++ pgrrsc.slcfg 
@@ -32,7 +32,7 @@ site @PITT { 
                nids    = supercell0-priv.frank.sam.pitt.edu, 
                          supercell0.frank.sam.pitt.edu; 
                fsroot  = /data/supercell0; 
-               self_test = "fshealthtest %r"; 
+               self_test = "fshealthtest -u 95 %r"; 
        } 
 
reinstall on supercell0
 
cd /opt/sam/slash2/src/inf/pgrrsc/ 
make instal 
 
you will want to restart sliod at some point as well
AttachmentSize
slash2.csh388 bytes
slash2.sh.txt387 bytes