HPC Storage
Along with access to the compute clusters we provide each research group with cluster storage space for research data. The storage is separated into three quotas: Home, Project, and 60-day Scratch. Each of these quotas limit both the amount in bytes and number of files you can store. Hitting your quota stops you from being able to write data, and can cause jobs to fail. You can monitor your storage usage by running the getquota command on a cluster. No sensitive data can be stored on any cluster storage, except for Milgram.
Backups
The only storage backed up on every cluster is Home. We do provide local snapshots, covering at least the last 2 days, on Home and Project directories (see below for details). Please see our HPC Policies page for additional information about backups.
Storage Spaces
For an overview of which filesystems are mounted on each cluster, see the HPC Resources documentation.
Home
Quota: 125 GiB and 500,000 files per person
Your home directory is where your sessions begin by default. Its intended use is for storing scripts, notes, final products (e.g. figures), etc. Its path is /home/netid (where netid is your Yale netid) on every cluster. Home storage is backed up daily. If you would like to restore files, please contact us with your netid and the list of files/directories you would like restored.
Project
Quota: 1 TiB and 5,000,000 files per group, expanded to 4 TiB on request
Project storage is shared among all members of a specific group. Project storage is not backed up, so we strongly recommend that you have a second copy somewhere off-cluster of any valuable data you have stored in project.
You can access this space through a symlink, or shortcut, in your home directory called project. See our Sharing Data documentation for instructions on sharing data in your project space with other users.
Project quotas are global to the whole project space, so if the group ownership on a file is your group, it will count towards your quota, regardless of its location within project. This can occasionally create confusion for users who belong to multiple groups and they need to be mindful of which files are owned by which of their group affiliations to ensure proper accounting.
Purchased Storage
Quota: varies
Storage purchased for the dedicated use by a single group or collection of groups provides similar functionality as project storage and is also not backed up.
See below for details on purchasing storage.
Purchased storage, if applicable, is assigned on one of our two storage systems:
- on the Gibbs filesystem in a
/gpfs/gibbs/pi/directory under the group's name - on the Palmer filesystem in a
/vast/palmer/pi/directory under the group's name
Unlike project space described above, all files in your purchased storage count towards your quotas, regardless of file ownership.
60-Day Scratch
Quota: 10 TiB and 15,000,000 files per group
60-day scratch is intended to be used for storing temporary data. Any file in this space older than 60 days will automatically be deleted. We send out a weekly warning about files we expect to delete the following week. Like project, scratch quota is shared by your entire research group. If we begin to run low on storage, you may be asked to delete files younger than 60 days old. Artificial extension of scratch file expiration is forbidden without explicit approval from the YCRC. Please purchase storage if you need additional longer term storage.
You can access this space through a symlink, or shortcut, in your home directory called palmer_scratch (or scratch60 on Milgram). See our Sharing Data documentation for instructions on sharing data in your scratch space with other users.
Check Your Usage and Quotas
To inspect your current usage, run the command getquota. Here is an example output of the command:
This script shows information about your quotas on grace.
If you plan to poll this sort of information extensively,
please contact us for help at hpc@yale.edu
## Usage Details for support (as of Jan 25 2023 12:00)
Fileset User Usage (GiB) File Count
---------------------- ----- ---------- -------------
gibbs:project ahs3 568 121,786
gibbs:project kln26 435 423,219
gibbs:project ms725 233 456,736
gibbs:project pl543 427 1,551,959
gibbs:project rdb9 1952 1,049,346
gibbs:project tl397 605 2,573,824
----
gibbs:pi_support ahs3 0 1
gibbs:pi_support kln26 5886 14,514,143
gibbs:pi_support ms725 19651 2,692,158
gibbs:pi_support pl543 328 142,936
gibbs:pi_support rdb9 1047 165,553
gibbs:pi_support tl397 175 118,038
## Quota Summary for support (as of right now [*palmer stats are gathered once a day])
Fileset Type Usage (GiB) Quota (GiB) File Count File Limit Backup Purged
---------------------- ------- ------------ ----------- ------------- ------------- --------- ---------
palmer:home.grace USR 63 125 216,046 500,000 Yes No
gibbs:project GRP 3832 10240 3,350,198 10,000,000 No No
palmer:scratch GRP 0 10240 903 15,000,000 No 60 days
gibbs:pi_support FILESET 27240 30720 17,647,694 22,000,000 No No
The per-user breakdown is only generated periodically, and the summary at the bottom is close to real-time. Purchased storage allocations will only appear in the getquota output for users who have data in that directory.
Purchase Additional Storage
For long-term allocations, additional project storage spaces can be purchased on one of our shared filesystems, which provides similar functionality to the primary project storage. This storage currently costs $280/TiB (minimum of 10 TiB, with exact pricing to be confirmed before a purchase is made). The price covers all costs, including administration, power, cooling, networking, etc. YCRC commits to making the storage available for 5 years from the purchase date, after which the storage allocation will need to be renewed, or the allocation will expire and be removed (see Storage Expiration Policy).
For shorter-term or smaller allocations, we have a monthly billing option. More details on this option can be found here (CAS login required).
Please note that, as with existing project storage, purchased storage will not be backed up, so you should make arrangements for the safekeeping of critical files off the clusters. Please contact us with your requirements and budget to start the purchasing process.
Purchased storage, as with all storage allocations, are subject to corresponding file count limit to preserve the health of the shared storage system. The file count limits for different size allocations are listed above. If you need additional files beyond your limit, contact us to discuss as increases may be granted on a case-by-case basis and at the YCRC's discretion.
| Allocation Quota | File Count Limit |
|---|---|
| < 50 TiB | 10 million |
| 50-99 TiB | 20 million |
| 100-499 TiB | 40 million |
| 500-999 TiB | 50 million |
| >= 1 PiB | 75 million |
HPC Storage Best Practices
Stage Data
Large datasets are often stored off-cluster on departmental servers, Storage@Yale, in cloud storage, etc.
If these data are too large to fit in your current quotas and you do not plan on purchasing more storage (see above), you must 'stage' your data.
Since the permanent copy of the data remains on off-cluster storage, you can transfer a working copy to palmer_scratch, for example.
Both Grace and McCleary have dedicated transfer partitions where you can submit long-running transfer jobs.
When your computation finishes you can remove the copy and transmit or copy results to a permanent location.
Please see the Staging Data documentation for more details and examples.
Prevent Large Numbers of Small Files
The parallel filesystems the clusters use perform poorly with very large numbers of small files. This is one reason we enforce file count quotas. If you are running an application that unavoidably make large numbers of files, do what you can to reduce file creation. Additionally you can reduce load on the filesystem by spreading the files across multiple subdirectories. Delete unneeded files between jobs and compress or archive collections of files.