PyTorch: Difference between revisions

PyTorch natively supports parallelizing work across multiple CPUs in two ways: Intra-op parallelism and Inter-op parallelism.

* '''Intra-op''' refers to PyTorch's parallel implementations of operators commonly used in Deep Learning, such as matrix multiplication and convolution, using [https://www.openmp.org OpenMP] directly or through low-level libraries like [https://en.wikipedia.org/wiki/Math_Kernel_Library MKL] and [https://www.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/api-based-programming/intel-oneapi-deep-neural-network-library-onednn.html OneDNN]. Whenever you run PyTorch code that performs such operations, they will automatically leverage multi-threading over as many CPU cores as are available to your job.

* '''Inter-op''' parallelism on the other hand refers to PyTorch's ability to execute different parts of your code concurrently. This modality of parallelism typically requires that you explicitly design your program such that different parts can run in parallel. Examples include code that leverages PyTorch's Just-In-Time compiler <code>torch.jit</code> to run asynchronous tasks in a [https://pytorch.org/docs/stable/jit.html#built-in-functions-and-modules TorchScript] program.