Gurobi: Difference between revisions

[http://www.gurobi.com/ Gurobi] is a commercial software suite for solving complex optimization problems.  This page wiki page describes the non-commercial use of Gurobi software on Compute Canada clusters and is currently a work in progress.

Compute Canada supports and provides a free license to use Gurobi on the [[Graham]], [[Cedar]], [[Beluga/en|Béluga]] and [[Niagara]] clusters.  The license provides a total number of 4096 simultaneous uses (tokens in use) and permits distributed optimization with up to 100 nodes.  A single user can run multiple simultaneous jobs.  In order to use Gurobi you must agree to certain conditions. Please [[Technical_support | contact support]] and include a copy of the following completed Academic Usage Agreement.  You will then be added into the Compute Canada license file as a permitted user within a few days:

You do NOT need to create a <code>~/.licenses/gurobi.lic</code> file.  The required settings to use the Compute Canada Gurobi license are configured by default when you load a gurobi module on any cluster.  To verify your username has been added to the Compute Canada Gurobi license and working properly run the following command:

If it returns "Success" then you can begin using gurobi immediately! If the test returns "Fail" then check whether a file named <i>~/.license/gurobi</i> exists, if so then rename or remove it, reload the module and try the test again.  If it still returns "Fail" then check whether there are any environment variables containing GUROBI defined in either of our your <i>~/.bashrc</i> or <i>~/.bash_profile</I> files.  If you find any then comment or remove those lines, logout and login again, reload the gurobi module and try the test again.  If you still get "Fail" then contact [[Technical_support | contact support]] for help.

Note that all Gurobi license checkouts are handled by a single license server located in Ontario, so that it is important for you to assure that your use of Gurobi limits as much as possible license checkout attempts.  Rather than checking out a license for each invocation of Gurobi in a job - which may occur dozens or even hundreds of times - you should ensure that your program, whatever the language or computing environment used, only makes a single license checkout and then reuses this license token throughout the lifetime of the job. This will improve your job's performance since contacting a remote license server is very costly in time and also improve the responsiveness of our license server for everyone who is using Gurobi.  <span style="color:red">Failure to use Gurobi carefully in this regard may ultimately result in random intermittent license checkout failures for all users.  If this happens you will be contacted and asked to kill all your jobs until your program is fixed and tested to ensure the problem is gone.</span>    Some documentation on this subject for C++ programs may be found [https://www.gurobi.com/documentation/9.5/refman/cpp_env2.html here], explaining how to create a single Gurobi environment which can then be used for all your models. Python users can consult this [https://www.gurobi.com/documentation/9.5/refman/py_env_start.html page], which discusses how to implement this same idea of using a single environment and thus a single license token with multiple models.  Other programs when run in parallel that call Gurobi such as R can also easily trigger the problem especially when many simultaneous jobs are submitted and/or run in parallel.

You can record API calls and repeat them with command

Once a slurm script has been prepared for a Gurobi problem it can be submitted to the queue by running the <code>sbatch script-name.sh</code> command.  The jobs status in the queue can then be checked by running the <code>sq</code> command.  The following slurm scripts demonstrate solving 2 problems provided in the <tt> examples</tt> directory of each gurobi module:

The following slurm script utilizes the [https://www.gurobi.com/documentation/9.5/quickstart_linux/solving_the_model_using_th.html Gurobi command-line interface] to solve a [https://www.gurobi.com/documentation/9.5/quickstart_linux/solving_a_simple_model_the.html simple coin production model] written in [https://www.gurobi.com/documentation/9.5/refman/lp_format.html LP format].  The last line demonstrates how [https://www.gurobi.com/documentation/9.5/refman/parameters.html parameters] can be passed directly to the Gurobi command-line tool <code>gurobi_cl</code> using simple command line arguments.  For help selecting which [https://www.gurobi.com/documentation/9.5/refman/parameters.html parameters] are best used for a particular problem and choosing optimal values refer to both the <i>Performance and Parameters</i> & <i>Algorithms and Search</I> sections found in the Gurobi [https://support.gurobi.com/hc/en-us/categories/360000840331-Knowledge-Base Knowledge Base] as well as the extensive online Gurobi [https://www.gurobi.com/documentation/ Documentation].

This is an example slurm script for solving a [https://www.gurobi.com/documentation/9.5/examples/a_list_of_the_grb_examples.html simple facility location model] using [https://www.gurobi.com/documentation/9.5/examples/facility_py.html Gurobi Python].  The example shows how-to set the threads  [https://www.gurobi.com/documentation/9.5/refman/parameters.html#sec:Parameters parameter] with a dynamically created [https://www.gurobi.com/documentation/9.5/quickstart_linux/using_a_grb_env_file.html gurobi.env] file when using the [https://www.gurobi.com/documentation/9.5/refman/python_parameter_examples.html Gurobi python interface].  This must be done in the working directory for each submitted job otherwise gurobi will (by default) start as many execute [https://www.gurobi.com/documentation/9.5/refman/threads.html#parameter:Threads threads] as there are physical cores on the compute node, instead of using the number of physical cores allocated to the job by slurm, potentially slowing down the job and negatively impacting other user jobs running on the same node.

Gurobi brings it's own version of Python but that one does not contain any 3rd-party Python packages except Gurobi.  In order to use Gurobi together with popular Python  

Before we start, we need to decide which combination of versions for Gurobi and Python to use.  For example compare the python versions supported by the gurobi 8.11 and 9.0.1 modules:

Now that the environment has been activated install the Python packages we want to use, in this case <code>pandas</code> for example:

Once created we can activate our Gurobi environment at any time with:

Various topics can be found by visiting [https://www.gurobi.com/resources/ Resources] then clicking [https://www.gurobi.com/resources/?category-filter=code-example Code and Modeling Examples] and finally [https://www.gurobi.com/resource/modeling-examples-using-the-gurobi-python-api-in-jupyter-notebook/ Optimization with Python – Jupyter Notebook Modeling Examples].  Alternatively visit [https://support.gurobi.com/ support.gurobi.com] and search on <I>Jupyter Notebooks</I>.   

A demo case of using Gurobi with Jupyter notebooks on Compute Canada systems can be found in this [https://youtu.be/Qk3Le5HBxeg?t=2310 video recording, i.e. at time 38:28].