Graham: Difference between revisions
No edit summary |
No edit summary |
||
| Line 14: | Line 14: | ||
<!--T:4--> | <!--T:4--> | ||
The parallel filesystem, interconnects and external persistent storage ([[National Data Cyberinfrastructure|NDC-Waterloo]]) will be the same as [[Cedar|Cedar's]]. There is a slightly different mix of compute nodes. | The parallel filesystem, interconnects and external persistent storage ([[National Data Cyberinfrastructure|NDC-Waterloo]]) will be the same as [[Cedar|Cedar's]]. There is a slightly different mix of compute nodes. | ||
====Attached Storage System==== <!--T:4--> | |||
<!--T:5--> | |||
{| class="wikitable sortable" | |||
|- | |||
| '''$HOME''' || | |||
Standard home directory<br /> | |||
Not allocated<br /> | |||
Small, standard quota<br /> | |||
Larger requests should be on $PROJECT | |||
|- | |||
| '''$SCRATCH<br />Parallel High-performance filesystem''' || | |||
DDN storage subsystem with approximately 4PB usable capacity for temporary (<code>/scratch</code>) storage.<br /> | |||
Aggregate performance of 35GB/s. Available to all nodes.<br /> | |||
Not allocated<br /> | |||
Purged - inactive data will be purged | |||
|- | |||
|'''$PROJECT<br />External persistent storage''' | |||
|| | |||
Provided by the [[National_Data_Cyberinfrastructure|NDC]].<br /> | |||
Available to compute nodes, but not designed for parallel I/O workloads.<br /> | |||
|} | |||
====High Performance Interconnect==== <!--T:19--> | |||
<!--T:20--> | |||
<!-- | |||
InfiniBand is to be used. | |||
--> | |||
<!--T:21--> | |||
A low-latency high-performance fabric connecting all nodes and temporary storage. | |||
<!--T:22--> | |||
The design of Graham is to support multiple simultaneous parallel jobs of up to 1024 cores in a fully non-blocking manner. | |||
<!-- | |||
For larger jobs the interconnect has a 2:1 blocking factor, i.e., even for jobs running on several thousand cores the Graham system provides a high-performance interconnect. | |||
--> | |||
====Node types and characteristics==== <!--T:5--> | ====Node types and characteristics==== <!--T:5--> | ||
| Line 35: | Line 74: | ||
<!--T:8--> | <!--T:8--> | ||
Compute Canada is not currently able to disclose the specific CPU | Compute Canada is not currently able to disclose the specific CPU models or specifications. | ||
<!--T:9--> | <!--T:9--> | ||
Revision as of 17:26, 12 January 2017
Graham (GP3)
GRAHAM is a heterogeneous cluster, suitable for a variety of workloads, and located at the University of Waterloo. It is named after Wes Graham, the first director of the Computing Centre at Waterloo. It was previously known as "GP3" and is still identified as such in the 2017 RAC documentation.
System evaluation is not yet completed as of November 2016. Anticipated specifications, based on Waterloo's RFP and bids received, include the following. This information is NOT GUARANTEED and might not be complete. It is provided for planning purposes.
The parallel filesystem, interconnects and external persistent storage (NDC-Waterloo) will be the same as Cedar's. There is a slightly different mix of compute nodes.
Attached Storage System
| $HOME |
Standard home directory |
| $SCRATCH Parallel High-performance filesystem |
DDN storage subsystem with approximately 4PB usable capacity for temporary ( |
| $PROJECT External persistent storage |
Provided by the NDC. |
High Performance Interconnect
A low-latency high-performance fabric connecting all nodes and temporary storage.
The design of Graham is to support multiple simultaneous parallel jobs of up to 1024 cores in a fully non-blocking manner.
Node types and characteristics
| "Base" compute nodes | 800 nodes | 16 cores/socket, 2 sockets/node, 128 GB of memory. |
| "Bigmem512" nodes | 24 nodes | 16 cores/socket, 2 sockets/node, 512 GB of memory. |
| "Bigmem3000" nodes | 3 nodes | 16 cores/socket, 4 sockets/node, 3 TB of memory. |
| "GPU" nodes | 160 nodes | 16 cores/socket, 2 sockets/node, 128 GB of memory, 2 NVIDIA P100 GPUs. |
| "Large" nodes | 56 nodes | 16 cores/socket, 2 sockets/node, 256 GB of memory. |
All of the above nodes will have local (on-node) storage ranging from 300GB to 960GB SATA SSD drives.
Compute Canada is not currently able to disclose the specific CPU models or specifications.
Graham will have GPU's, but Compute Canada is not currently able to disclose specific details. The RFP used NVIDIA K80 as a baseline specification.
The completed system is expected to have around 35,000 cores.
The delivery and installation schedule is not yet confirmed.