cc_staff
353
edits
AI and Machine Learning: Difference between revisions
Jump to navigation
Jump to search
Marked this version for translation
(Tips about data storage) |
(Marked this version for translation) |
||
| Line 54: | Line 54: | ||
Compute Canada provides a wide range of storage options to cover the needs of our very diverse users. These storage solutions range from high-speed temporary local storage to different kinds of long-term storage, so you can choose the storage medium that best corresponds to your needs and usage patterns. Please refer to our documentation on [[Storage and file management]]. | Compute Canada provides a wide range of storage options to cover the needs of our very diverse users. These storage solutions range from high-speed temporary local storage to different kinds of long-term storage, so you can choose the storage medium that best corresponds to your needs and usage patterns. Please refer to our documentation on [[Storage and file management]]. | ||
<!--T:34--> | |||
Here are some tips: | Here are some tips: | ||
<!--T:35--> | |||
* If your dataset is around 10 GB* or below, it can probably fit in memory, depending on how much memory your job has. You should not read the data from disk during your machine learning task. | * If your dataset is around 10 GB* or below, it can probably fit in memory, depending on how much memory your job has. You should not read the data from disk during your machine learning task. | ||
* If your dataset is around 100 GB* or below, it can fit in the local storage of the compute node; please transfer it there at the beginning of the job. A temporary directory is available for each job at $SLURM_TMPDIR. An example is given in [[Tutoriel_Apprentissage_machine/en|our tutorial]]. A caveat of local node storage is that another job might be using it fully, leaving you no space (we currently studying this problem). | * If your dataset is around 100 GB* or below, it can fit in the local storage of the compute node; please transfer it there at the beginning of the job. A temporary directory is available for each job at $SLURM_TMPDIR. An example is given in [[Tutoriel_Apprentissage_machine/en|our tutorial]]. A caveat of local node storage is that another job might be using it fully, leaving you no space (we currently studying this problem). | ||