HPC File System
From UF HPC Wiki
Contents |
Home Directories
All home directories are currently located on a one Terabyte partition. As such, there is not a lot of space for files for all of our users. It is recommended that you use this space only for programs that you are developing, and keep large data sets and output from programs in the scratch space we provide.
Space Usage
At the HPC Center, we ask that space usage on home directories be kept to a minimum. We are in the process of implementing quotas, and we ask that users keep their usage on the home directories under 10gb.
Use of space on the scratch system is unlimited, but we ask that once you have completed a job and generated the data that you need, you remove the data and keep the space usage on the scratch system as small as possible. We do not want to limit your usage on this filesystem, but we do ask that you be responsible with your space usage.
To check and see how much space you are taking up in a specific directory, you can do the following:
$ du -h 28K ./test/amber/pbsa_dmp 24K ./test/amber/pbsa_ion 368K ./test/amber/pbsa_pgb 752K ./test/amber/pbsa_trx 952K ./test/amber/pheTI . . . $ du -hs 2.6G .
Transferring Files
Transferring files to or from the HPC file system can be done in two different ways, through the use of Secure Copy or through GridFTP.
Secure Copy
Unix Based Operating Systems
When using a Unix based operating system (linux, Solaris, etc.) transferring files is quite easy through the use of a program called scp. The program can be used from either end of the transfer stream, either submit.hpc.ufl.edu or your own machine. The basic form of the command is as follows:
$ scp [user@host]:file [user@host]:file
What the above means is that you can use the program to copy a file from the host that you are currently on to a remote location, or copy a file from a remote location to your current host.
Example: Copy from local to remote
The following would copy a file from the current directory of your own machine to your home directory on submit.hpc.ufl.edu. When the command executes, it would also ask for your password on submit.
$ scp example.txt user@submit.hpc.ufl.edu:~/example.txt
Example: Copy from remote to local
The following would copy a file from your home directory on submit to your local directory:
$ scp user@submit.hpc.ufl.edu:~/example.txt ./example.txt
Options
Some options that might be useful:
- -p
- Preserves modification times, access times, and modes from the original file.
- -r
- Recursively copy entire directories.
Microsoft Windows Based
There are a number of different programs available to provide secure file transfers. The product documentation for them should be read to fully understand how to transfer files.
GridFTP
This section has not yet been written.
Scratch Space
The UF HPC center has about 35tb of scratch space for use. This scratch space is located here:
/scratch/ufhpc
To create your own area on the scratch space, do the following:
[test@submit ~]$ cd /scratch/ufhpc/ [test@submit ufhpc]$ mkdir test [test@submit ufhpc]$ cd test [test@submit test]$ pwd /scratch/ufhpc/test [test@submit test]$
The above created the directory /scratch/ufhpc/test which has permissions set to your account. You can then use this space for job input/output.
Automounted Directories
The directories in /apps are all autofs directories, which means that they are not visible unless you explicitly type in their names...
[root@submit apps]# cd /apps/ [root@submit apps]# ls [root@submit apps]# cd matlab [root@submit matlab]# ls 2007a 2008a [root@submit matlab]# cd .. [root@submit apps]# ls matlab [root@submit apps]#
So, as we see above, the /apps directory appears to be empty at first, but as soon as I explicitly enter the matlab directory, that directory is visible.
