Best Practices on HPC Storage
User Quota Limits and the myquota command
All users have quote limits set on HPC fie systems. There are several types of quota limits, such as limits on the amount of disk space (disk quota), number of files (inode quota) etc. The default user quota limits on HPC file systems are listed on our Data Management page.
One of the common issues users report is running out of inodes in their home directory. This usually occurs during software installation, for example installing conda environment under their home directory. Running out of quota causes a variety of issues such as running user jobs being interrupted or users being unable to finish the installation of packages under their home directory.
Users can check their current utilization of quota using the myquota command. The myquota command provides a report of the current quota limits on mounted file systems, the user's quota utilization, as well as the percentage of quota utilization. For details, please see myquota
Large number of small files
In case your dataset or workflow requires to use large number of small files, this can create a bottleneck due to read/write rates. Please refer to our page on working with a large number of files to learn about some of the options we recommend to consider.
Installing Python packages
Your home directory is limited to a relatively small number of inodes (30,000). Creating conda/python environments in you home directory, this can eat easily exhaust your inode quota.
Please review the Package Management section of the Torch Software Page.
Troubleshooting quota issues
Being over your quota can cause problems logging into OOD and other issues. If you find that you're suddenly unable to log in, one of the first things you should check is your quota with the myquota command.
If you find that you're over your quota, myquota will show you if you are over your quota for space or i-nodes or both. You can then see which folders most contribute to you being over quota with the following commands:
Here's how you can see which folders contain the most files:
du --inodes -h -s -- * .[!.]* ..?* 2>/dev/null | sort -hr | head -n 5
and here's how you can see which ones most contribute to your disk usage:
du -h -s -- * .[!.]* ..?* 2>/dev/null | sort -hr | head -n 5
Moving common dot directories from /home to /scratch
As mentioned elsewhere in our documentation, a problem we often see for our users is that of filling their /home directories by using common tools like VSCode, Cursor, etc. These applications put a dot directory in your /home directory and it can quickly overwhelm your quota.
Our recommended fix for this is to make a directory for these files in your /scratch directory and then make a symbolic link in your /home that points to the one you just created in /scratch. This allows these applications to write to what appears to be your /home directory, but is actually stored in your /scratch directory. This can be accomplished with the following commands:
The first part of the following commands will remove the contents of the dot directory in your ${HOME} directory. Be sure to copy anything you'd like to save from that directory before executing.
For VSCode:
rm -rf ${HOME}/.vscode-server && mkdir -p ${SCRATCH}/vscode-server && ln -s ${SCRATCH}/vscode-server ${HOME}/.vscode-server
For Cursor:
rm -rf ${HOME}/.cursor-server && mkdir -p ${SCRATCH}/cursor-server && ln -s ${SCRATCH}/cursor-server ${HOME}/.cursor-server
For RStudio:
rm -rf ${HOME}/.local/share/rstudio && mkdir -p ${SCRATCH}/.local/share/rstudio && ln -s ${SCRATCH}/.local/share/rstudio ${HOME}/.local/share/rstudio
For details about changing your default RStudio directory please see User state storage.
Please contact hpc@nyu.edu if you have any questions.