Skip to content

Transfer Data

For all transfer methods you should already have set up your account on the cluster(s) you want to tranfer data to/from.

Data Transfer Nodes

Each cluster (except Milgram) has a dedicated node specially networked for high speed transfers both on and off-campus using the Yale Science Network. These transfer nodes host of our Globus service, enabling resilient transfers at scale. After logging in to a cluster, you can ssh to its transfer node with:

ssh transfer

Tip

If you are running a large transfer without Globus, run it inside a tmux session on the transfer node. This protects your transfer from network interruptions between your computer and the transfer node.

You may also use the transfer node to transfer data from your local machine using one of the below methods. From off-cluster, the nodes are accessible at the following hostnames. You must still be on-campus or on the VPN to access the transfer nodes.

Cluster Transfer Node
Grace transfer-grace.hpc.yale.edu
Farnam transfer-farnam.hpc.yale.edu
Ruddle transfer-ruddle.hpc.yale.edu
Milgram use login node, milgram.hpc.yale.edu

Graphical Transfer Tools

OOD Web Transfers

On Farnam, Grace, and Ruddle you can use their respective Open OnDemand sites to transfer files. This works best for small numbers of relatively small files. You can also directly edit scripts through this interface, alleviating the need to transfer scripts to your computer to edit.

MobaXterm (Windows)

MobaXterm is an all-in-one graphical client for Windows that includes a transfer pane for each cluster you connect to. Once you have established a connection to the cluster, click on the "Sftp" tab in the left sidebar to see your files on the cluster. You can drag-and-drop data into and out of the SFTP pane to upload and download, respectively.

Cyberduck

You can also transfer files between your local computer and a cluster using an FTP client, such as Cyberduck (OSX/Windows). You will need to configure the client with your netid as the username, the cluster transfer node as the hostname and your private key as the authentication method. An example configuration of Cyberduck is shown below.

Cyberduck sample configuration.

Cyberduck on Ruddle

Ruddle requires Multi-Factor Authentication so there are a couple additional configuration steps. Under Cyberduck > Preferences > Transfers > General change the setting to "Use browser connection" instead of "Open multiple connections".

When you connect type one of the following when prompted with a "Partial authentication success" window.

  • "push" to receive a push notification to your smart phone (requires the Duo mobile app)
  • "sms" to receive a verification passcode via text message
  • "phone" to receive a phone call

Large Transfers (Globus)

Globus is a web-enabled GridFTP service that transfers large datasets fast, securely, and reliably between computers configured to be endpoints. We have configured endpoints for most of the Yale clusters and you can configure your computer to transfer files as well. Please see our Globus page for Yale-specific documentation and their official docs to get started.

Command-Line Transfer Tools

scp and rsync (macOS/Linux/Linux on Windows)

Linux and macOS users can use scp or rsync. Use the hostname of the cluster transfer node (see above) to transfer files.

Note that you must have your ssh keys properly setup to use the commands outlined below. See the Cluster Access documentation for more info.

scp and sftp are both used from a Terminal window. The basic syntax of scp is

scp [from] [to]

The from and to can each be a filename or a directory/folder on the computer you are typing the command on or a remote host (e.g. the transfer node).

Transfer a File from Your Computer to a Cluster

Using the example netid abc123, following is run on your computer's local terminal.

scp myfile.txt abc123@transfer-grace.hpc.yale.edu:/home/fas/admins/abc123/test

In this example, myfile.txt is copied to the directory /home/fas/admins/abc123/test: on Grace. This example assumes that myfile.txt is in your current directory. You may also specify the full path of myfile.txt.

scp /home/xyz/myfile.txt abc123@transfer-grace.hpc.yale.edu:/home/fas/admins/abc123/test

Transfer a Directory to a Cluster

scp -r mydirectory abc123@transfer-grace.hpc.yale.edu:/home/fas/admins/abc123/test

In this example, the contents of mydirectory are transferred. The -r indicates that the copy is recursive.

Transfer Files from the Cluster to Your Computer

Assuming you would like the files copied to your current directory:

scp abc123@transfer-grace.hpc.yale.edu:/home/fas/admins/abc123/myfile.txt .

Note that . represents your current working directory. To specify the destination, simply replace the . with the full path:

scp abc123@transfer-grace.hpc.yale.edu:/home/fas/admins/abc123/myfile.txt /path/myfolder

Rclone

To move data to and from cloud storage (Box, Dropbox, Wasabi, AWS S3, or Google Cloud Storage, etc.) , we recommend using Rclone. It is installed as a module on all of the clusters and can be installed on your computer. You will need to configure it for each kind of storage you would like to transfer to with:

rclone configure

You'll be prompted for a name for the connection (e.g mys3), and then details about the connection. Once you've saved that configuration, you can use that connection name to copy files with similar syntax to scp and rsync:

rclone copy localpath/myfile mys3:bucketname/
rclone sync localpath/mydir mys3:bucketname/remotedir

We recommend that you protect your configurations with a password. You'll see that as an option when you run rclone config. For all the Rclone documentaion please refer to the official site.


Last update: May 7, 2020