Python: Difference between revisions

Note that the multiprocessing module is restricted to using a single compute node, so the speedup achievable by your program is usually limited to the total number of CPU cores in that node.  If you want to go beyond this limit and use multiple nodes, consider using mpi4py or [[Apache Spark/en#PySpark|PySpark]].  Other methods of parallelizing Python (not all of them necessarily supported on our clusters) are listed [https://wiki.python.org/moin/ParallelProcessing here]. Also note that you can greatly improve  the performance of your Python program by ensuring it is written efficiently, so that should be done first before parallelizing.  If you are not sure if your Python code is efficient, please contact [[technical support]] and have them look at your code.