Python: Difference between revisions

The <code>pip</code> command can install packages from a variety of sources, including PyPI and pre-built distribution packages called Python [https://pythonwheels.com/ wheels]. Compute Canada provides Python wheels for a number of packages. In the above example, the [https://pip.pypa.io/en/stable/reference/pip_wheel/#cmdoption-no-index <code>--no-index</code>] option tells <code>pip</code> to ''not'' install from PyPI, but instead to install only from locally-available packages, i.e. the Compute Canada wheels.

Whenever a Compute Canada wheel is available for a given package, we strongly recommend to use it by way of the <code>--no-index</code> option. Compared to using packages from PyPI, wheels that have been compiled by Compute Canada staff can prevent issues with missing or conflicting dependencies, and were optimised for our clusters hardware and libraries. See [[#Available_wheels|Available wheels]].

If you omit the <code>--no-index</code> option, <code>pip</code> will search both PyPI and local packages, and use the latest version available. If PyPI has a newer version, it will be installed instead of the Compute Canada wheel, possibly causing issues. If you are certain that you prefer to download a package from PyPI rather than use a wheel, you can use the <code>--no-binary</code> option, which tells <code>pip</code> to ignore pre-built packages entirely. Note that this will also ignore wheels that are distributed through PyPI, and will always compile the package from source.

In some cases, such as TensorFlow, Compute Canada provides wheels for a specific host (cpu or gpu), suffixed with <tt>_cpu</tt> or <tt>_gpu</tt>. Packages dependent on <tt>tensorflow</tt> will then fail to install.  

Note that the above instructions require all of the packages you need to be available in the python wheels that we provide (see "Available wheels" below). If the wheel is not available in our wheelhouse, you can pre-download it (see "Pre-downloading packages" section below). If you think that the missing wheel should be included in the Compute Canada wheelhouse, please contact [[Technical support]] to make a request.

Note that the multiprocessing module is restricted to using a single compute node, so the speedup achievable by your program is usually limited to the total number of CPU cores in that node.  If you want to go beyond this limit and use multiple nodes, consider using mpi4py or [[Apache Spark/en#PySpark|PySpark]].  Other methods of parallelizing Python (not all of them necessarily supported on Compute Canada clusters) are listed [https://wiki.python.org/moin/ParallelProcessing here]. Also note that you can greatly improve  the performance of your Python program by ensuring it is written efficiently, so that should be done first before parallelizing.  If you are not sure if your Python code is efficient, please contact [[technical support]] and have them look at your code.