NCI has installed an additional 814 compute nodes on Raijin through the support of the Australian Government. Each compute node consists of 2 x Intel Xeon E5-2690 v4 (14-core, 2.6GHz) Broadwell processors. The new system will appear on Raijin as separate queues (normalbw, expressbw and hugemem), where "bw" stands for Intel Broadwell architecture. The compute nodes in the system utilise a Mellanox EDR interconnect (100Gb/s) in 2:1 blocking fat-tree topology, and there is a total of 2016Gb/s (252GB/s) of bandwidth to Raijin's core network (e.g. for access to storage).
The specifications of these new nodes (a total of 22,792 additional cores) are:
|Memory||JobFS (Local Disk)||Cores||Base Frequency||Max Turbo Frequency|
|536||128GB||440GB SSD||28||2.6 GHz||3.5 GHz|
|268||256GB||440GB SSD||28||2.6 GHz||3.5 GHz|
|10||1TB||440GB SSD||28||2.6 GHz||3.5 GHz|
Maximum turbo frequency is not always achievable. Please see http://www.intel.com/technology/turboboost/ for further information.
While applications for Raijin's original compute nodes will run on these nodes, in order to make the most of the newer architecture these nodes provides, we strongly encourage you to rebuild your application using the newer instruction sets. When doing so, for the Intel compilers you should specify
-xCORE-AVX2 while for the GNU compilers you should use
For example, if you are compiling the code for Broadwell nodes on Raijin's login nodes (which are Xeon Sandy Bridge nodes), you would use a command similar to this:
We always recommend using the latest version of the Intel compilers, as older versions may not be able to optimise for newer architectures.
To submit jobs to these nodes you should select the appropriate queue for your job.
|Queue||Memory Available||Priority||Charge Rate|
|normalbw||128GB / 256GB||Regular||1.25 SU / core / hour|
|expressbw||128GB / 256GB||High||3.75 SU / core / hour|
|hugemem||1TB||Regular||1.25 SU / core / hour|
You should specify the queue you wish to use via the
-q option to
qsub, or with a
#PBS -q directive in your PBS job script.
As with the normal and express queues, any job larger than one node must request CPUs in multiples of full nodes. This means that you should consider the number of CPUs required for your job to ensure you are requesting an appropriate amount for your job. In particular where you may previously have requested 16 CPUs you should now look at requesting 28 CPUs.