Translations:Tutoriel Apprentissage machine/34/en: Difference between revisions

Jump to navigation Jump to search

Translations:Tutoriel Apprentissage machine/34/en (view source)

Revision as of 18:34, 8 October 2019

35 bytes added ,  5 years ago
no edit summary
No edit summary
No edit summary
Line 1: Line 1:
The filesystems on Compute Canada clusters are designed for a small number of extremely large files. Make sure that the data set which you need for your training is an archive format like ''tar'', which you can then transfer to your job's compute node when the job starts. '''If you do not respect these rules, you risk causing enormous numbers of I/O operations on the shared filesystem, leading to performance issues on the cluster for all of its users.''' If you want to learn more about how to handle collections of large number of files, we recommend that you spend some time reading [[Handling_large_collections_of_files|this page]].
The shared storage on Compute Canada clusters are not designed to handle lots of small files (they are optimized for very large files). Make sure that the data set which you need for your training is an archive format like ''tar'', which you can then transfer to your job's compute node when the job starts. '''If you do not respect these rules, you risk causing enormous numbers of I/O operations on the shared filesystem, leading to performance issues on the cluster for all of its users.''' If you want to learn more about how to handle collections of large number of files, we recommend that you spend some time reading [[Handling_large_collections_of_files|this page]].
Lemc2220
cc_staff
353

edits

Retrieved from "https://docs.alliancecan.ca/wiki/Special:MobileDiff/77018"

Navigation menu