GNU Parallel

Introduction

GNU Parallel is a tool for running many sequential tasks at the same time on one or more nodes. It is useful for running a large number of sequential tasks, especially if they are short or variable duration, as well as when doing a parameter sweep. We will only cover the basic options here, for more advanced usage, please see the official documentation.

By default, parallel will run as many tasks as the number of cores on the node, therefore maximizing resource usage. When a task finishes, a new task will automatically be started by parallel.

Basic Usage

Parallel uses curly brackets {} as parameters for the command to be run. For example, to run zip on all the text files in a directory, you can execute

[name@server ~]$ ls *.txt | parallel zip {}

An alternative syntax is to use :::, such as this example:

[name@server ~]$ parallel echo {} ::: $(seq 1 3)
1
2
3

Multiple Arguments

You can also use multiple arguments by enumerating them, for example:

[name@server ~]$ parallel echo {1} {2} ::: $(seq 1 3) ::: $(seq 2 3)
1 2
1 3
2 2
2 3
3 2
3 3

Content of a file as argument list

The syntax :::: takes the content of a file to generate the list of values for the arguments. For example, if you have a list of parameter values in the file mylist.txt, you may display its content with:

[name@server ~]$ parallel echo {1} :::: mylist.txt