38,907
edits
Translations:Python/60/en: Difference between revisions
Jump to navigation
Jump to search
Importing a new version from external source
(Importing a new version from external source) |
(Importing a new version from external source) |
||
| Line 1: | Line 1: | ||
Note that the multiprocessing module is restricted to using a single compute node, so the speedup achievable by your program is usually limited to the total number of CPU cores in that node. If you want to go beyond this limit and use multiple nodes, consider using mpi4py or [[Apache Spark/en#PySpark|PySpark]]. Other methods of parallelizing Python (not all of them necessarily supported on our clusters) are listed [https://wiki.python.org/moin/ParallelProcessing here]. Also note that you can greatly improve the performance of your Python program by ensuring it is written efficiently, so that should be done first before parallelizing. If you are not sure if your Python code is efficient, please contact [[technical support]] and have them look at your code. | Note that the <code>multiprocessing</code> module is restricted to using a single compute node, so the speedup achievable by your program is usually limited to the total number of CPU cores in that node. If you want to go beyond this limit and use multiple nodes, consider using mpi4py or [[Apache Spark/en#PySpark|PySpark]]. Other methods of parallelizing Python (not all of them necessarily supported on our clusters) are listed [https://wiki.python.org/moin/ParallelProcessing here]. Also note that you can greatly improve the performance of your Python program by ensuring it is written efficiently, so that should be done first before parallelizing. If you are not sure if your Python code is efficient, please contact [[technical support]] and have them look at your code. | ||