Irreproducible data eg. source code
Large data I/O; data maintained beyond one job
Processing of large data files
Archiving large data files
external – access using the mdss command
Yes: 2 copies in two different locations
I/O intensive data over the job's lifetime
local to each individual Raijin node
duration of job
unigrp– determined by their host institution, and
projectid(s)– one for each project they are attached to.
Please make sure you specify #PBS -lother=gdata1 when submitting jobs accessing files in /g/data1. If /g/data1 filesystem is not available, your job will not start. The following command can be used to monitor the status of /g/data1 on Raijin and can be incorporated inside your jobscript:
/opt/rash/bin/modstatus -n gdata1_status
Please make sure you specify #PBS -lother=mdss when submitting jobs accessing files in MDSS. If the MDSS filesystem is not available, your job will not start. The following command can be used to monitor the status of MDSS on Raijin and can be incorporated inside your jobscript:
/opt/rash/bin/modstatus -n mdss_status
Users request allocation of /jobfs as part of their job submission – the actual disk quota for a particular job is given by the jobfs request. Requests larger than 420GB for Sandybridge (copyq, normal, express), 700 GB for the GPU queue, 400GB for everything else (knl, normalbw, expressbw, normalsp, hugemem, gpupascal) will be automatically redirected to /short (but will still be deleted at the end of the job).
lquotaon Raijin to see your disk quota and usage, and see the Disk Quota Policy document for details of the ramifications of exceeding the quotas.
/short/projectidon each compute system. Users connected to the project have
rwxpermissions in that directory and so may create their own files in those areas.
nci_accounton each machine to see your disk quota and usage. See the Disk Quota Policy below for details of the ramifications of exceeding the quotas.
nci_account. An excessive number of inodes causes a number of filesystem problems, hence the limit.
Lots of small I/O to
To achieve good I/O performance, try to read or write binary files in large chunks (of around 1MB or greater). To find out more details of how to best tune your I/O contact us.
#PBS -lother=gdataN(where N=1, 2 or 3 depending on which /g/data filesystem is being used) when submitting jobs. If
/g/dataNfilesystem is not available, your job will not start. The following command can be used to monitor the status of
/g/dataNon Raijin and can be incorporated inside your jobscript:
/opt/rash/bin/modstatus -n gdataN_status
For more information on how to use the above filesystems (/home, /short, and /g/data) see Lustre Basics and Lustre Best Practices.
netcp -tcommand option below.)
/massdata/projectidon that system. This path CANNOT be directly accessed from Raijin login.
Remote access to your massdata directory is by the
mdss utility or the
netmv commands (see man
mdss/netcp/netmv for full details.) The
mdss commands operate on files in that remote directory.
mdss put – copy files to the MDSS get – copy files from the MDSS mk/rmdir – create/delete directories on the MDSS ls – list directories
netcp/netmv netmv and netcp generate a script, then submit a batch request to PBS to copy files (and directories) from Raijin to the MDSS. In the case of netmv, remove the files from Raijin if the copy has succeeded. -t create a tarfile to transfer -z/-Z gzip/compress the file to be transferred Please use at least the -t option if you wish to archive a directory structure of numerous small files.
rwxpermissions in that directory and so may create their own files in those areas.
Note: always use
nci_accounton the compute machines to see your MDSS quota and usage. See the Disk Quota Policy below for details of the ramifications of exceeding the quotas.
mdssaccess is intended for relatively modest mass data storage needs. Users with larger capacity storage or more sophisticated access needs should contact us to get an account on the data cluster.
How do I transfer files between massdata and my local machine?
Please note that massdata is not designed for small files. Attempting to store or retrieve files less than a few megabytes will result in extremely poor performance for all users. If you wish to store lots of small files to massdata, please use a utility such as tar to combine them into a single, larger file.
To transfer files between massdata and your local machine, we recommend this two-step workflow.
-ljobfs=5GBrequests 5 GB. Use integers and units of MB or GB (not case-sensitive). The maximum request must be less than or equal to the local disc storage of the node/s
Your batch job can access its jobfs via the environment variable
PBS_JOBFS. Jobs spanning multiple nodes with local JOBFS space on each node should use the
/opt/pbs/bin/pbsdsh -N ... command in the batch script to act on all JOBFS directories, e.g.
/opt/pbs/bin/pbsdsh -N ls $PBS_JOBFS
For example, if you want local copies of files generated before the current batch run you can do the following to make them available on each nodes' jobfs area.
/opt/pbs/bin/pbsdsh -N cp original_file $PBS_JOBFS
Don’t put any quotes around the command issued under pbsdsh.
It is not possible to use the
Users who are dealing with large files in large chunks (i.e. > 1 MB reads and writes) have a number of options available to them to improve their I/O performance. Contact us for assistance in choosing the best options.
As well as the generally available filesystems listed above, there may be high performance filesystems, utilities or techniques available to improve the I/O performance of your workload. Please contact us if you think this may be relevant to you.
To avoid the disk usage of one or two users or projects adversely affecting other users, we have implemented a policy of checking disk usage on a regular basis and taking the actions detailed below.
View your disk quotas and current usages on Raijin by running the commands:
nci_account for mdss, /short and /g/data usage
lquota for /home, /short and /g/data usage
Administrative disk usage limits
Administrative disk usage limits are imposed on the file systems /home, /short, /g/data and the mdss file storage area.
Users should ensure they have enough space available before starting jobs. Note that adjustments to quotas, on a temporary or a long-term basis, will be made given reasonable justification.
The consequences of a project or user exceeding these administrative disk usage limits are outlined in the following table:
Hard Limit Exceeded
Once quota exceeded, new files will not be able to write to the /home filesystem. No email reminder is currently implemented.
Except for the copyq, queued jobs will not start. Currently running jobs will be allowed to continue. The copyq will remain available to allow archiving. All members of the project will be sent an email to this effect once a day. Queued jobs will be released when the quota system next detects that the disk usage is back under the quota limit. This should occur within 30 minutes of the project getting back under the disk quota limit. If the usage exceeds 90% of the quota, all members of the project will be sent an email to this effect once a month as a reminder. This will not affect the queue until the usage reaches 100% of the limit.
Except for the copyq, queued jobs will not start. Currently running jobs will be allowed to continue. The copyq will remain available to allow archiving. All members of the project will be sent an email to this effect. Due to the relative “expense” of reassessing file space usage under MDSS, reactivation of your queued job will not automatically occur for up to 24 hours after the removal of files. Contact firstname.lastname@example.org to request manual reactivation.
It is important to understand that these limits are not the native filesystem quotas available on most Unix filesystems – the behaviour and actions described above are quite different. Exceeding administrative limits should not cause job failures nor stop users from managing files etc. However users should be aware that real filesystem “hard quotas” are also imposed but with a limit much higher than the administrative limits discussed above. If a user or project reaches the relevant filesystem hard quota, jobs are likely to fail due to write operations failing.
Note: There are two conventions for defining digital storage capacities, namely, base 2 (1KB = 1024 bytes) and base 10 (1KB = 1000 bytes), and both are in common use. At NCI, the following definitions are used when referring to digital storage capacities (filesystem quota limits, usage, etc.):