Cedar

Cedar (GP2)

CEDAR is a heterogeneous cluster, suitable for a variety of workloads, located at Simon Fraser University. It is named for the Western Red Cedar, B.C.’s official tree and of great spiritual significance to the region's First Nations people. It was previously known as "GP2" and is still identified as such in the 2017 RAC documentation.

System evaluation is not yet completed as of November 2016. Anticipated specifications, based on SFU's RFP and bids received, include the following. This information is not guaranteed and might not be complete. It is provided for planning purposes.

Attached Storage System

$HOME	Standard home directory Not allocated Small, standard quota Larger requests should be on $PROJECT
$SCRATCH Parallel High-performance filesystem	DDN storage subsystem with approximately 4PB usable capacity for temporary (/scratch) storage. Aggregate performance of 35GB/s. Available to all nodes. Not allocated Purged - inactive data will be purged
$PROJECT External persistent storage	Provided by the NDC. Available to compute nodes, but not designed for parallel I/O workloads.
High performance interconnect	Intel OmniPath (version 1) interconnect. A low-latency high-performance fabric connecting all nodes and temporary storage. The design of Cedar is to support multiple simultaneous parallel jobs of at least 1024 cores in a fully non-blocking manner. Jobs larger than 1024 cores would be less well-suited for the topology.

Node types and characteristics:

Cedar will have a total of 27,696 CPU cores for computation, and 584 GPU devices. Total theoretical peak double precision performance is 936 tetaflops for CPUs, plus 2,744 for GPUs, yielding over 3.6 petaflops of theoretical peak double precision performance. 22 fully connected "islands" of 32 base or large nodes will each have 1024 cores in a fully non-blocking topology, with each island expected to yield over 30 teraflops of measured double precision performance (measured with high performance LINPACK).

"Base" compute nodes:	576 nodes	128 GB of memory, 16 cores/socket, 2 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4.
"Large" compute nodes:	128 nodes	256 GB of memory, 16 cores/socket, 2 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4.
"Bigmem500"	24 nodes	0.5 TB (512 GB) of memory, 16 cores/socket, 2 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4.
"Bigmem1500" nodes	24 nodes	1.5 TB of memory, 16 cores/socket, 2 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4.
"GPU base" nodes:	114 nodes	128 GB of memory, 12 cores/socket, 2 sockets/node, 4 NVIDIA P100 Pascal GPUs/node (12GB HBM2 memory), 2 GPUs/PCI root. Intel "Broadwell" CPUs at 2.2Ghz, model E5-2650 v4
"GPU large" nodes.	32 nodes	256 GB of memory, 12 cores/socket, 2 sockets/node, 4 NVIDIA P100 Pascal GPUs/node (12GB HBM2 memory), All GPUs on the same PCI root. E5-2650 v4
"Bigmem3000" nodes	4 nodes	3 TB of memory, 8 cores/socket, 4 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E7-4809 v4.

All of the above nodes will have local (on-node) storage.

Compute Canada is not currently able to disclose the specific GPU model or specifications. The RFP used NVIDIA K80 as a baseline specification.

The completed system is expected to have over 25,000 cores, 900 nodes, and 500 GPUs.

The delivery and installation schedule is not yet known, but the procurement team has confidence that the system will be in production for the start of the allocations year on April 1, 2017.