Graham/en: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
(Updating to match new version of source page)
(Updating to match new version of source page)
Line 7: Line 7:
GRAHAM is a heterogeneous cluster, suitable for a variety of workloads, and located at the University of Waterloo. It is named after [https://en.wikipedia.org/wiki/Wes_Graham Wes Graham], the first director of the Computing Centre at Waterloo. It was previously known as "GP3" and is still identified as such in the [https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions/ 2017 RAC] documentation.
GRAHAM is a heterogeneous cluster, suitable for a variety of workloads, and located at the University of Waterloo. It is named after [https://en.wikipedia.org/wiki/Wes_Graham Wes Graham], the first director of the Computing Centre at Waterloo. It was previously known as "GP3" and is still identified as such in the [https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions/ 2017 RAC] documentation.


System evaluation is not yet completed as of November 2016.  Anticipated specifications, based on Waterloo's RFP and bids received, include the following. This information is NOT GUARANTEED and might not be complete.  It is provided for planning purposes.
The parallel filesystem and external persistent storage ([[National Data Cyberinfrastructure|NDC-Waterloo]]) are similar to [[Cedar|Cedar's]]. The interconnect is different and there is a slightly different mix of compute nodes.


The parallel filesystem, interconnects and external persistent storage ([[National Data Cyberinfrastructure|NDC-Waterloo]]) will be the same as [[Cedar|Cedar's]]. There is a slightly different mix of compute nodes.
====Attached storage systems====
 
{| class="wikitable sortable"
|-
| '''$HOME''' ||
Standard home directory<br />
Small, standard quota<br />
Larger requests should be on $PROJECT
|-
| '''$SCRATCH<br />Parallel high-performance filesystem''' ||
Approximately 3PB usable capacity for active or temporary (<code>/scratch</code>) storage.<br />
Available to all nodes.<br />
Not allocated<br />
Purged - inactive data will be purged
|-
|'''$PROJECT<br />External persistent storage'''
||
Provided by the [[National_Data_Cyberinfrastructure|NDC]].<br />
Available to compute nodes, but not designed for parallel I/O workloads.<br />
|}
 
====High-performance interconnect====
 
Low-latency high-bandwidth Infiniband fabric connects all nodes and scratch storage.
 
The design of Graham is to support multiple simultaneous parallel jobs of up to 1024 cores in a fully non-blocking manner.
<!--
For larger jobs the interconnect has a 2:1 blocking factor, i.e., even for jobs running on several thousand cores the Graham system provides a high-performance interconnect.
-->


====Node types and characteristics====
====Node types and characteristics====
''Processor type:'' All nodes have Intel E5-2683 V4 CPUs, running at 2.1 GHz
''GPU type:'' P100 12g


{| class="wikitable sortable"
{| class="wikitable sortable"
|-
|-
| "Base" compute nodes || Over 500 nodes || 128 GB of memory each
| "Base" compute nodes || 800 nodes || 16 cores/socket, 2 sockets/node, 128 GB of memory
|-
|-
| "Large" compute nodes || Over 100 nodes || 256 GB of memory each
| "Large" nodes || 56 nodes || 16 cores/socket, 2 sockets/node, 256 GB of memory.
|-
|-
| "Bigmem512" nodes|| TBD || 512 GB of memory each
| "Bigmem500" nodes|| 24 nodes || 16 cores/socket, 2 sockets/node, 512 GB of memory.
|-
|-
| "Bigmem3000" nodes || TBD || 3 TB of memory each
| "Bigmem3000" nodes || 3 nodes || 16 cores/socket, 4 sockets/node, 3 TB of memory.
|-
| "GPU" nodes || 160 nodes || 16 cores/socket, 2 sockets/node, 128 GB of memory, 2 NVIDIA P100 GPUs.
|}
|}


All of the above nodes will have local (on-node) storage.
All of the above nodes will have approximately 1TB of local (on-node) storage provided by SSD drives available in /tmp.
 
Compute Canada is not currently able to disclose the specific CPU model or specifications.
 
Graham will have GPU's, but Compute Canada is not currently able to disclose specific details. The RFP used NVIDIA K80 as a baseline specification.
 
The completed system is expected to have over 25,000 cores.


The delivery and installation schedule is not yet known.
The delivery and installation schedule is not yet confirmed.


<noinclude>
<noinclude>
</noinclude>
</noinclude>

Revision as of 15:52, 26 January 2017

Other languages:


Graham (GP3)

GRAHAM is a heterogeneous cluster, suitable for a variety of workloads, and located at the University of Waterloo. It is named after Wes Graham, the first director of the Computing Centre at Waterloo. It was previously known as "GP3" and is still identified as such in the 2017 RAC documentation.

The parallel filesystem and external persistent storage (NDC-Waterloo) are similar to Cedar's. The interconnect is different and there is a slightly different mix of compute nodes.

Attached storage systems

$HOME

Standard home directory
Small, standard quota
Larger requests should be on $PROJECT

$SCRATCH
Parallel high-performance filesystem

Approximately 3PB usable capacity for active or temporary (/scratch) storage.
Available to all nodes.
Not allocated
Purged - inactive data will be purged

$PROJECT
External persistent storage

Provided by the NDC.
Available to compute nodes, but not designed for parallel I/O workloads.

High-performance interconnect

Low-latency high-bandwidth Infiniband fabric connects all nodes and scratch storage.

The design of Graham is to support multiple simultaneous parallel jobs of up to 1024 cores in a fully non-blocking manner.

Node types and characteristics

Processor type: All nodes have Intel E5-2683 V4 CPUs, running at 2.1 GHz

GPU type: P100 12g

"Base" compute nodes 800 nodes 16 cores/socket, 2 sockets/node, 128 GB of memory
"Large" nodes 56 nodes 16 cores/socket, 2 sockets/node, 256 GB of memory.
"Bigmem500" nodes 24 nodes 16 cores/socket, 2 sockets/node, 512 GB of memory.
"Bigmem3000" nodes 3 nodes 16 cores/socket, 4 sockets/node, 3 TB of memory.
"GPU" nodes 160 nodes 16 cores/socket, 2 sockets/node, 128 GB of memory, 2 NVIDIA P100 GPUs.

All of the above nodes will have approximately 1TB of local (on-node) storage provided by SSD drives available in /tmp.

The delivery and installation schedule is not yet confirmed.


Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=Graham/en&oldid=7850"