Rclone
rclone is a command line tool to sync files and directories to and from all major cloud storage sites. You can use rclone to sync files and directories between Yale clusters and Yale Box, google drive, etc. The following instructions cover basics to setup and use rclone on Yale clusters. For more information about Rclone, please visit its website at https://rclone.org.
Set up Rclone on YCRC clusters
Before accessing a remote cloud storage using rclone, you need to run rclone config to configure
the storage for rclone. Since rclone config will try to bring up a browser for you to authorize
the cloud storage, we recommend you to use Open OnDemand.
To run rclone config on OOD, first click Remote Desktop from the OOD dashboard.
Once a session starts running, click Connect to Remote Desktop and
you will see a terminal on the desktop in the browser.
Run rclone config at the command line of the terminal.
During configuration, you will see a message similar to the following:
If your browser does not open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Tip
If you received an error stating that your session has expired for DUO, simply paste the link and reload the page. If you still get the expired message, log out of CAS in your browser by going to https://secure.its.yale.edu/cas/logout. After logging out, paste the link and reload.
Examples
The following examples show you how to set up rclone for Google Drive and Yale Box. In both examples, we name our remote cloud storage as 'remote' in the configuration. You can provide any name you want.
The example below is a screen dump when setting up rclone on Farnam for Google Drive.
[pl543@c03n06 ~]$ rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / 1Fichier
\ "fichier"
2 / Alias for an existing remote
\ "alias"
[...]
15 / Google Drive
\ "drive"
[...]
42 / seafile
\ "seafile"
Storage> 15
** See help for drive backend at: https://rclone.org/drive/ **
Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").
client_id>
OAuth Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Full access all files, excluding Application Data Folder.
\ "drive"
2 / Read-only access to file metadata and file contents.
\ "drive.readonly"
/ Access to files created by rclone only.
3 | These are visible in the drive website.
| File authorization is revoked when the user deauthorizes the app.
\ "drive.file"
/ Allows read and write access to the Application Data folder.
4 | This is not visible in the drive website.
\ "drive.appfolder"
/ Allows read-only access to file metadata but
5 | does not allow any access to read or download file content.
\ "drive.metadata.readonly"
scope> 1
ID of the root folder
Leave blank normally.
Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.
Enter a string value. Press Enter for the default ("").
root_folder_id>
Service Account Credentials JSON file path
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
Enter a string value. Press Enter for the default ("").
service_account_file>
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n> y
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=6glRr_mpEORxHevlOaaYyw
Log in and authorize rclone for access
Waiting for code...
Got code
Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No (default)
y/n> n
--------------------
[remote]
type = drive
scope = drive
token = {"access_token":"ya29.A0ArdaM-mBYFKBE2gieODvdANCZRV6Y8QHhQF-lY74E9fr1HTLOwwLRuoQQbO9P-Jdip62YYhqXfcuWT0KLKGdhUb9M8g2Z4XEQqoNLwZyA-FA2AAYYBqB","token_type":"Bearer","refresh_token":"1//0dDu3r6KVakgYIARAAGA0NwF-L9IrWIuG7_f44x-uLR2vvBocf4q-KnQVhlkm18TO2Fn0GjJp-cArWfj5kY84","expiry":"2021-02-25T17:28:18.629507046-05
:00"}
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
remote drive
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
The example below is a screen dump when setting up rclone on Farnam for Yale Box.
[pl543@c14n07 ~]$ rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / 1Fichier
\ "fichier"
[...]
6 / Box
\ "box"
[...]
Storage> box
** See help for box backend at: https://rclone.org/box/ **
Box App Client Id.
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_id>
Box App Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> y
If your browser does not open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
--------------------
[remote]
type = box
token = {"access_token":"PjIXHUZ34VQSmeUZ9r6bhc9ux44KMU0e","token_type":"bearer","refresh_token":"VumWPWP5Nd0M2C1GyfgfJL51zUeWPPVLc6VC6lBQduEPsQ9a6ibSor2dvHmyZ6B8","expiry":"2019-10-21T11:00:36.839586736-04:00"}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
remote box
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
Tip
if you want to use rclone for a shared google drive, you should answer 'y' when it asks whether you want to configure it as a Shared Drive.
Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No (default)
y/n> y
Tip
rclone config creates a file storing cloud storage configurations for rclone.
You can check the file name with rclone config file. The config file can be
copied to other clusters so that you can use rclone on the other clusters without running rclone config again.
Use Rclone on Yale clusters
As we have used remote as the name of the cloud storage in our examples above,
we will continue using it in the following examples.
You should replace it with the actual name you have picked up for the cloud storage in your configuration.
Tip
If you forgot the name of the cloud storage you have configured, run rclone config show and the name will be shown in [].
$ rclone config show
[remote]
type = drive
scope = drive
token = {"access_token":"mytoken","expiry":"2021-07-09T22:13:56.452750648-04:00"}
root_folder_id = myid
List files
rclone ls remote:/
Copy files
# to download a file to the cluster
rclone copy remote:/path/to/filename .
# to upload a file from the cluster to the cloud storage
rclone copy filename remote:/path/to/
Help
rclone help