Graham

Expected availability: In production. RAC 2017's implemented June 30, 2017

Login node: graham.computecanada.ca

Globus endpoint: computecanada#graham-dtn

Data mover node (rsync, scp, cp,...): gra-dtn1.computecanada.ca

GRAHAM is a heterogeneous cluster, suitable for a variety of workloads, and located at the University of Waterloo. It is named after Wes Graham, the first director of the Computing Centre at Waterloo. It was previously known as "GP3" and is still identified as such in the 2017 RAC documentation.

The parallel filesystem and external persistent storage (NDC-Waterloo) are similar to Cedar's. The interconnect is different and there is a slightly different mix of compute nodes.

The Graham system is sold and supported by Huawei Canada, Inc. It is entirely liquid cooled, using rear-door heat exchangers.

Getting started with Graham

How to run jobs

Transfering data

Attached storage systems

Home space	Location of home directories. Each home directory has a small, fixed quota. Not allocated via RAS or RAC. Larger requests go to Project space. Has daily backup.
Scratch space 3.6PB total volume Parallel high-performance filesystem	For active or temporary (/scratch) storage. Not allocated. Large fixed quota per user. Inactive data will be purged.
Project space External persistent storage	Part of the National Data Cyberinfrastructure. Allocated via RAS or RAC. Not designed for parallel I/O workloads. Use Scratch space instead. Large adjustable quota per project. Has daily backup.

High-performance interconnect

Mellanox FDR (56Gb/s) and EDR (100Gb/s) InfiniBand interconnect. FDR is used for GPU and cloud nodes, EDR for other node types. A central 324-port director switch aggregates connections from islands of 1024 cores each for CPU and GPU nodes. The 56 cloud nodes are a variation on CPU nodes, and are on a single larger island sharing 8 FDR uplinks to the director switch.

A low-latency high-bandwidth Infiniband fabric connects all nodes and scratch storage.

Nodes configurable for cloud provisioning also have a 10Gb/s Ethernet network, with 40Gb/s uplinks to scratch storage.

The design of Graham is to support multiple simultaneous parallel jobs of up to 1024 cores in a fully non-blocking manner.

For larger jobs the interconnect has a 8:1 blocking factor, i.e., even for jobs running on multiple islands the Graham system provides a high-performance interconnect.

Graham high performance interconnect diagram

Node types and characteristics

A total of 35,520 cores and 320 GPU devices, spread across 1,107 nodes of different types.

Processor type: All nodes except bigmem3000 have Intel E5-2683 V4 CPUs, running at 2.1 GHz

GPU type: P100 12g

base nodes	864 nodes	128 GiB of memory (125 Gib usable), 16 cores/socket, 2 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4. 960GB SATA SSD.
large nodes (cloud configuration)	56 nodes	256 GiB of memory (251 GiB usable), 16 cores/socket, 2 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4. 960GB SATA SSD.
GPU nodes	160 nodes	128 GiB of memory (125 Gib usable), 16 cores/socket, 2 sockets/node, 2 NVIDIA P100 Pascal GPUs/node (12GB HBM2 memory). Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4. 1.6TB NVMe SSD.
bigmem500 nodes	24 nodes	512 GiB of memory (503 GiB usable), 16 cores/socket, 2 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4. 960GB SATA SSD.
bigmem3000 nodes	3 nodes	3 TiB of memory (3023 GiB usable), 16 cores/socket, 4 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E7-4850 v4. 960GB SATA SSD.

Best practice for local on-node storage is to use the temporary directory generated by Slurm, $SLURM_TMPDIR. Note that this directory and its contents will disappear upon job completion.

Resource Allocation Competition

Globus is a file transfer service