National systems

Sharing: Public

New Systems

Author:CC Migration team

Compute

Overview

GP2 and GP3 are almost identical systems with some minor differences in the actual mix of large memory, small memory and GPU nodes.

Name	Description	Approximate Capacity	Availability
CC-Cloud Resources (GP1)	Cloud	7,640 cores	In production (integrated with west.cloud)
GP2	heterogeneous, general-purpose cluster (Serial and small parallel jobs) Small cloud partition	25,000 cores	April, 2017
GP3	heterogeneous, general-purpose cluster (Serial and small parallel jobs) Small cloud partition	25,000 cores	April, 2017
LP	a cluster designed for large parallel jobs	60,000 cores	Late 2017

Note that GP1, GP2 and LP will all have large, high-performance attached storage.

GP2 (SFU)

The GP2 system evaluation is not yet completed (as of early November 2016). Anticipated specifications, based on SFU's RFP and bids received, include the following. This information is not guaranteed and might not be complete. It is provided for planning purposes.

Parallel filesystem:	Approximately 4PB usable capacity for temporary (/scratch) storage. Aggregate performance of approximately 40GB/s. Available to all nodes.
External persistent storage:	10PB or more of storage for persistent (/project and other) storage. Available to compute nodes, but not designed for parallel workloads.
High performance interconnect:	Low-latency high-performance fabric connecting all nodes and temporary storage. The design of GP2 is to support multiple simultaneous parallel jobs of at least 1024 cores in a fully non-blocking manner. Jobs larger than 1024 cores would be less well-suited for the topology.

Node types and characteristics:

"Base" compute nodes:	Over 500 nodes	128GB of memory.
"Large" compute nodes:	Over 100 nodes	256GB of memory.
"Bigmem512" and "Bigmem1500" nodes	24 nodes	each with 512GB and 1.5TB of memory respectively.

All of the above nodes will have 16 cores/socket (32 cores/node), with an anticipated frequency of 2.1Ghz.

"GPU base" nodes:	Over 100 nodes with 4 GPUs each.	12 cores/socket (24 cores/node) with an anticipated frequency of 2.2Ghz. GPUs on a dual PCI root.
"GPU large" nodes.	Approximately 30 nodes	same configuration as "GPU base," but a single PCI root.
"Bigmem3000" nodes	4 nodes	each with 3TB of memory. These are 4-socket nodes with 8 cores/socket.

All of the above nodes will have local (on-node) storage.

Login, Gateway and Data Transfer nodes

Compute Canada is not currently able to disclose the specific GPU model or specifications. The RFP used NVIDIA K80 as a baseline specification.

The total GP2 system is anticipated to have over 25,000 cores, 900 nodes, and 500 GPUs.

The delivery and installation schedule is not yet known, but the procurement team has confidence that the system will be in production for the start of the allocations year on April 1, 2017.

GP3 (Waterloo)

P3 system evaluation is not yet completed (as of early November 2016). Anticipated specifications, based on Waterloo's RFP and bids received, include the following. This information is NOT GUARANTEED and might not be complete. It is provided for planning purposes.

The parallel filesystem, interconnects and external persistent storage will be the same as GP2.

Node types and characteristics:

"Base" compute nodes:	Over 500 nodes	128GB of memory.
"Large" compute nodes:	Over 100 nodes	256GB of memory.
"Bigmem512" nodes	TBD	each with 512GB
"Bigmem3000" nodes	TBD	3 TB

GPU nodes

All of the above nodes will have local (on-node) storage.

Login, Gateway and Data Transfer Nodes

Compute Canada is not currently able to disclose the specific GPU or CPU model or specifications. The RFP used NVIDIA K80 as a baseline specification.

The total GP3 system is anticipated to have over 20,000 cores.

The delivery and installation schedule is not yet known.

National Data Cyberinfrastructure (NDC)

Type	Location	Capacity	Availability	Comments
Nearline (Long-term) File Storage	SFU and Waterloo NDC-SFU NDC-Waterloo	15 PB each	Late Autumn 2016	Tape backup Silo replacement Aiming for redundant tape backup across sites
Object Store	All 4 sites NDC-Object	Small to start (A few PB usable)	Late 2017	Fully distributed, redundant object storage Accessible anywhere Allows for redundant, high availability architectures S3 and File interfaces New service aimed at experimental and observational data
Special Purpose	All 4 sites	~3.5 PB	Specialized migration plans	Special purpose for Atlas, LHC, SNO+, CANFAR dCache and other customized configurations

Note that due to Silo decommissioning Dec.31/2016 it will be necessary to provide interim storage while the NDC is developed. See Migration2016:Silo for details.