For all transfer methods you should already have set up your account on the cluster(s) you want to tranfer data to/from.
Data Transfer Nodes
Each cluster (except Milgram) has a dedicated node specially networked for high speed transfers both on and off-campus using the Yale Science Network. These transfer nodes host of our Globus service, enabling resilient transfers at scale. After logging in to a cluster, you can ssh to its transfer node with:
If you are running a large transfer without Globus, run it inside a tmux session on the transfer node. This protects your transfer from network interruptions between your computer and the transfer node.
You may also use the transfer node to transfer data from your local machine using one of the below methods. From off-cluster, the nodes are accessible at the following hostnames. You must still be on-campus or on the VPN to access the transfer nodes.
|Milgram||use login node,
Graphical Transfer Tools
OOD Web Transfers
On Farnam, Grace, and Ruddle you can use their respective Open OnDemand sites to transfer files. This works best for small numbers of relatively small files. You can also directly edit scripts through this interface, alleviating the need to transfer scripts to your computer to edit.
MobaXterm is an all-in-one graphical client for Windows that includes a transfer pane for each cluster you connect to. Once you have established a connection to the cluster, click on the "Sftp" tab in the left sidebar to see your files on the cluster. You can drag-and-drop data into and out of the SFTP pane to upload and download, respectively.
You can also transfer files between your local computer and a cluster using an FTP client, such as Cyberduck (OSX/Windows). You will need to configure the client with your netid as the username, the cluster transfer node as the hostname and your private key as the authentication method. An example configuration of Cyberduck is shown below.
Cyberduck on Ruddle
Ruddle requires Multi-Factor Authentication so there are a couple additional configuration steps. Under
Cyberduck > Preferences > Transfers > General change the setting to "Use browser connection" instead of "Open multiple connections".
When you connect type one of the following when prompted with a "Partial authentication success" window.
- "push" to receive a push notification to your smart phone (requires the Duo mobile app)
- "sms" to receive a verification passcode via text message
- "phone" to receive a phone call
Large Transfers (Globus)
Globus is a web-enabled GridFTP service that transfers large datasets fast, securely, and reliably between computers configured to be endpoints. We have configured endpoints for most of the Yale clusters and you can configure your computer to transfer files as well. Please see our Globus page for Yale-specific documentation and their official docs to get started.
Command-Line Transfer Tools
scp and rsync (macOS/Linux/Linux on Windows)
Note that you must have your ssh keys properly setup to use the commands outlined below. See the Cluster Access documentation for more info.
scp and sftp are both used from a Terminal window. The basic syntax of
scp [from] [to]
The from and to can each be a filename or a directory/folder on the computer you are typing the command on or a remote host (e.g. the transfer node).
Transfer a File from Your Computer to a Cluster
Using the example netid
abc123, following is run on your computer's local terminal.
scp myfile.txt email@example.com:/home/fas/admins/abc123/test
In this example,
myfile.txt is copied to the directory
/home/fas/admins/abc123/test: on Grace. This example assumes that
myfile.txt is in your current directory. You may also specify the full path of
scp /home/xyz/myfile.txt firstname.lastname@example.org:/home/fas/admins/abc123/test
Transfer a Directory to a Cluster
scp -r mydirectory email@example.com:/home/fas/admins/abc123/test
In this example, the contents of
mydirectory are transferred. The
-r indicates that the copy is recursive.
Transfer Files from the Cluster to Your Computer
Assuming you would like the files copied to your current directory:
scp firstname.lastname@example.org:/home/fas/admins/abc123/myfile.txt .
. represents your current working directory.
To specify the destination, simply replace the
. with the full path:
scp email@example.com:/home/fas/admins/abc123/myfile.txt /path/myfolder
To move data to and from cloud storage (Box, Dropbox, Wasabi, AWS S3, or Google Cloud Storage, etc.) , we recommend using Rclone. It is installed as a module on all of the clusters and can be installed on your computer. You will need to configure it for each kind of storage you would like to transfer to with:
You'll be prompted for a name for the connection (e.g mys3), and then details about the connection. Once you've saved that configuration, you can use that connection name to copy files with similar syntax to
rclone copy localpath/myfile mys3:bucketname/ rclone sync localpath/mydir mys3:bucketname/remotedir
We recommend that you protect your configurations with a password. You'll see that as an option when you run rclone config. For all the Rclone documentaion please refer to the official site.