Transfer protocols

A number of methods allow transferring data in and out of PSMN computing center. For most cases, we recommend using SSH-based file transfer commands, such as scp, sftp, or rsync. They will provide the best performance for data transfers from and to computing center.

SCP (Secure Copy)

The easiest command to use to transfer files to/from PSMN is scp. It works like the cp command, except it work over the network to copy files from one computer to another, using SSH protocol.

For instance, the following command will copy the file named myfile from my local machine to the mydir directory in my home directory on PSMN:

scp myfile mylogin@allo-psmn.psmn.ens-lyon.fr:~/mydir/

(replace mylogin by your login as provided by PSMN).

You can copy myfile under a different name, or to another directory, with the following commands:

scp myfile mylogin@allo-psmn.psmn.ens-lyon.fr:~/inputfile
scp myfile mylogin@allo-psmn.psmn.ens-lyon.fr:~/mydir/subdir/foofile

To copy back files from PSMN to your local machine, you just need to reverse the order of the arguments, as in this example:

scp mylogin@allo-psmn.psmn.ens-lyon.fr:~/inputfile local_inputfile

scp also support recursive copying of directories, with -r option:

scp -r mydir/ mylogin@allo-psmn.psmn.ens-lyon.fr:~/

SFTP (Secure File Transfer Protocol)

SFTP clients are interactive file transfer programs (as to FTP), which perform all operations over an encrypted transport.

A variety of graphical SFTP clients are available:

When setting up your connection to PSMN in the above, use these informations:

host: allo-psmn.psmn.ens-lyon.fr
port: 22
username: your login at PSMN
password: your password at PSMN

OpenSSH also provide a command-line SFTP, named sftp, which can take advantage of ssh-agent and ssh keys. Example of use:

$ sftp mylogin@allo-psmn.psmn.ens-lyon.fr
Connected to allo-psmn.psmn.ens-lyon.fr.
sftp>

There are many tutorials online containing more informations about SFTP clients. Here’s one.

rsync

If you have complex hierarchies of files to transfer, or if you need to synchronize a set of files and directories between your local machine and PSMN storages, rsync will be one of the best tools to do the job. It will efficiently transfer and synchronize files across systems, by checking the timestamp and size of files. Which means that it won’t re-transfer files that have not changed since the last transfer, and will complete faster.

Also, if, for any reason, a transfer is interrupted, you might end up with part of files being transferred. Rather than restarting the transfer from scratch, rsync will only transfer what needs to be transferred: missing files, modified files, etc.

For instance, to transfer the whole ~/test/ folder tree from my local machine to my home directory on PSMN, I can use the following command:

$ rsync -avzP -e ssh ~/test/ mylogin@allo-psmn.psmn.ens-lyon.fr:~/test

Refer to the rsync manual for more options. Here’s another tutorial.

Unison

unison is a file-synchronization tool that is available on PSMN clusters.

See Unison homepage for more.

SSHFS

Sometimes, moving files in and out of the cluster, and maintaining two copies of each of the files you work on, both on your local machine and on PSMN, may be painful. Fortunately, PSMN offers the ability to mount its home filesystem to your local machine, using a secure and encrypted connection (and vice-versa, if your workstation expose a SSH server).

With SSHFS, a FUSE-based filesystem implementation used to mount remote SSH-accessible filesystems, you can access your files on PSMN as if they were locally stored on your own computer.

Hint

Be aware that, while very convenient, SSHFS is also very slow, due to FUSE.

This comes particularly handy when you need to access those files from an application that is not available on PSMN, but that you already use or can install on your local machine. Like a data processing program that you have licensed for your own computer but can’t be use on PSMN, a specific text editor that only runs on MacOS, or any data-intensive 3D rendering software that wouldn’t work comfortably enough over a forwarded X11 connection (See also Visualization server).

SSHFS is available for all platforms (Linux, MacOS and Windows).

Warning

SSHFS on MacOS

SSHFS on macOS is known to try to automatically reconnect filesystem mounts after resuming from sleep or suspend, even without any valid credentials. As a result, it will generate a lot of failed connection attempts and likely make your IP address blacklisted on ssh.psmn.ens-lyon.fr or allo-psmn.psmn.ens-lyon.fr.

Make sure to unmount your SSHFS drives before putting your macOS system to sleep to avoid this situation.

For instance, on a Linux machine with SSHFS installed, you could mount your PSMN home directory with the following commands:

$ mkdir ~/PSMN_home
$ sshfs mylogin@allo-psmn.psmn.ens-lyon.fr:~/ ~/PSMN_home

(replace mylogin by your login as provided by PSMN).

And to unmount it:

$ umount ~/PSMN_home

or:

$ fusermount -u ~/PSMN_home

For more information about using SSHFS on your local machine, you can refer to this tutorial for more details and examples.