Graham

Graham (GP3)

Expected availability: 2017 May 31 for opportunistic use

Login node: graham.computecanada.ca

Globus endpoint: computecanada#graham

GRAHAM is a heterogeneous cluster, suitable for a variety of workloads, and located at the University of Waterloo. It is named after Wes Graham, the first director of the Computing Centre at Waterloo. It was previously known as "GP3" and is still identified as such in the 2017 RAC documentation.

The parallel filesystem and external persistent storage (NDC-Waterloo) are similar to Cedar's. The interconnect is different and there is a slightly different mix of compute nodes.

The Graham system is sold and supported by Huawei Canada, Inc. It is entirely liquid cooled, using rear-door heat exchangers.

Attached storage systems

$HOME	Standard home directory Not allocated Small, standard quota Larger requests should be on $PROJECT
$SCRATCH Parallel high-performance filesystem	Huawei OceanStor-based storage subsystem with approximately 3.6PB usable capacity for active or temporary (/scratch) storage. Aggregate performance of approximately 30GB/s. Available to all nodes. Not allocated Purged - inactive data will be purged
$PROJECT External persistent storage	Provided by the NDC. Available to compute nodes, but not designed for parallel I/O workloads.

High-performance interconnect

Mellanox FDR (56Gb/s) and EDR (100Gb/s) InfiniBand interconnect. FDR is used for GPU and cloud nodes, EDR for other node types. A central 324-port director switch aggregates connections from islands of 1024 cores each for CPU and GPU nodes. The 60 cloud nodes are a variation on CPU nodes, and are on a single larger island sharing 8 FDR uplinks to the director switch.

A low-latency high-bandwidth Infiniband fabric connects all nodes and scratch storage.

Nodes configurable for cloud provisioning also have a 10Gb/s Ethernet network, with 40Gb/s uplinks to scratch storage.

The design of Graham is to support multiple simultaneous parallel jobs of up to 1024 cores in a fully non-blocking manner.

For larger jobs the interconnect has a 8:1 blocking factor, i.e., even for jobs running on multiple islands the Graham system provides a high-performance interconnect.

Graham high performance interconnect diagram

Node types and characteristics

Processor type: All nodes except bigmem3000 have Intel E5-2683 V4 CPUs, running at 2.1 GHz

GPU type: P100 12g

"Base" compute nodes	800 nodes	128 GB of memory, 16 cores/socket, 2 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4. 1.2TB NVME SSD.
"Large" nodes (cloud configuration)	56 nodes	256 GB of memory, 16 cores/socket, 2 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4. 1.2TB NVME SSD.
"Bigmem500" nodes	24 nodes	0.5 TB (512 GB) of memory, 16 cores/socket, 2 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4. 1.2TB NVME SSD.
"Bigmem3000" nodes	3 nodes	3 TB of memory, 14 cores/socket, 4 sockets/node. Intel "Haswell" CPUs at 2.2Ghz, model E7-4850 v3. 1.2TB NVME SSD.
"GPU" nodes	160 nodes	128 GB of memory, 12 cores/socket, 2 sockets/node, 2 NVIDIA P100 Pascal GPUs/node (12GB HBM2 memory). Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4. 800GB SATA SSD.

Local (on-node) storage in the above nodes will be available as /tmp.

Globus is a file transfer service