cc_staff
176
edits
Python: Difference between revisions
tone down the advocacy a bit and note that this works for single-node jobs.
(Stop suggesting Python 2) |
(tone down the advocacy a bit and note that this works for single-node jobs.) |
||
| Line 103: | Line 103: | ||
=== Creating virtual environments inside of your jobs === <!--T:36--> | === Creating virtual environments inside of your jobs === <!--T:36--> | ||
Parallel filesystems such as the ones used on our clusters are very good at reading or writing large chunks of data, but | Parallel filesystems such as the ones used on our clusters are very good at reading or writing large chunks of data, but can be bad for intensive use of small files. Launching a software and loading libraries, such as starting python and loading a virtual environment, can be slow for this reason. | ||
As a workaround for this kind of slowdown, and especially for single-node Python jobs, you can create your virtual environment inside of your job, using the compute node's local disk. It may seem counter-intuitive to recreate your environment for every job, but it can be faster than running from the parallel filesystem, and will give you some protection against some filesystem performance issues. This can be achieved using the following submission script example : | |||
<!--T:37--> | <!--T:37--> | ||
| Line 169: | Line 171: | ||
<!--T:43--> | <!--T:43--> | ||
Note that the above instructions require all of the packages you need to be available in the python wheels that we provide (see "Listing available wheels" below). If the wheel is not available in our wheelhouse, you can pre-download it (see "Pre-downloading packages" section below). If you think that the missing wheel should be included in the Compute Canada wheelhouse, please contact [[Technical support]] to make a request. | Note that the above instructions require all of the packages you need to be available in the python wheels that we provide (see "Listing available wheels" below). If the wheel is not available in our wheelhouse, you can pre-download it (see "Pre-downloading packages" section below). If you think that the missing wheel should be included in the Compute Canada wheelhouse, please contact [[Technical support]] to make a request. | ||
=== Listing available wheels === <!--T:23--> | === Listing available wheels === <!--T:23--> | ||