Gurobi: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
mNo edit summary
No edit summary
 
(47 intermediate revisions by 3 users not shown)
Line 6: Line 6:
[http://www.gurobi.com/ Gurobi] is a commercial software suite for solving complex optimization problems.  This wiki page describes the non-commercial use of Gurobi software on our clusters.
[http://www.gurobi.com/ Gurobi] is a commercial software suite for solving complex optimization problems.  This wiki page describes the non-commercial use of Gurobi software on our clusters.


== License Limitations == <!--T:2-->
== License limitations == <!--T:2-->


<!--T:3-->
<!--T:3-->
We support and provide a free license to use Gurobi on the [[Graham]], [[Cedar]], [[Beluga/en|Béluga]] and [[Niagara]] clusters.  The license provides a total number of 4096 simultaneous uses (tokens in use) and permits distributed optimization with up to 100 nodes.  A single user can run multiple simultaneous jobs.  In order to use Gurobi you must agree to certain conditions. Please [[Technical_support | contact support]] and include a copy of the following completed Academic Usage Agreement.  You will then be added into our license file as a permitted user within a few days.
We support and provide a free license to use Gurobi on the [[Graham]], [[Cedar]], [[Beluga/en|Béluga]] and [[Niagara]] clusters.  The license provides a total number of 4096 simultaneous uses (tokens in use) and permits distributed optimization with up to 100 nodes.  A single user can run multiple simultaneous jobs.  In order to use Gurobi, you must agree to certain conditions. Please [[Technical_support | contact support]] and include a copy of the following completed agreement.  You will then be added into our license file as a user within a few days.


===Academic Usage Agreement=== <!--T:4-->
===Academic usage agreement=== <!--T:4-->


<!--T:41-->
<!--T:41-->
My Alliance username is "_______" and I am a member of the academic institution "_____________________". This message confirms that I will only use the Gurobi license provided on Digital Research Alliance of Canada systems for the purpose of non-commercial research project(s) to be published in publicly available article(s).
My Alliance username is "_______" and I am a member of the academic institution "_____________________."  This message confirms that I will only use the Gurobi license provided on Digital Research Alliance of Canada systems for the purpose of non-commercial research project(s) to be published in publicly available article(s).


===Configuring your account=== <!--T:43-->
===Configuring your account=== <!--T:43-->
Line 24: Line 24:


<!--T:44-->
<!--T:44-->
If it returns "Success" you can begin using Gurobi immediately.  If the test returns "Fail" then check whether a file named <i>~/.license/gurobi</i> exists; if so, rename or remove this file, reload the module and try the test again.  If it still returns "Fail", check whether there are any environment variables containing GUROBI defined in either of our your <i>~/.bashrc</i> or <i>~/.bash_profile</I> files.  If you find any, comment or remove those lines, log out and log in again, reload the Gurobi module and try the test again.  If you still get "Fail", contact [[Technical_support | contact support]] for help.
If it returns "Success" you can begin using Gurobi immediately.  If the test returns "Fail" then check whether a file named <i>~/.license/gurobi</i> exists; if so, rename or remove this file, reload the module and try the test again.  If it still returns "Fail", check whether there are any environment variables containing GUROBI defined in either of our your <i>~/.bashrc</i> or <i>~/.bash_profile</I> files.  If you find any, comment or remove those lines, log out and log in again, reload the Gurobi module and try the test again.  If you still get "Fail", [[Technical_support | contact support]] for help.


===Minimizing License Checkouts=== <!--T:216-->
===Minimizing license checkouts=== <!--T:216-->


<!--T:217-->
<!--T:217-->
Note that all Gurobi license checkouts are handled by a single license server located in Ontario; it is therefore important for you to make sure that your use of Gurobi limits as much as possible license checkout attempts.  Rather than checking out a license for each invocation of Gurobi in a job--which may occur dozens or even hundreds of times--you should ensure that your program, whatever the language or computing environment used, only makes a single license checkout and then reuses this license token throughout the lifetime of the job. This will improve your job's performance since contacting a remote license server is very costly in time and will also improve the responsiveness of our license server for everyone who is using Gurobi.  <span style="color:red">Failure to use Gurobi carefully in this regard may ultimately result in random intermittent license checkout failures for all users.  If this happens, you will be contacted and asked to kill all your jobs until your program is fixed and tested to ensure the problem is gone.</span>   Some documentation on this subject for C++ programs may be found [https://www.gurobi.com/documentation/9.5/refman/cpp_env2.html here], explaining how to create a single Gurobi environment which can then be used for all your models. Python users can consult this [https://www.gurobi.com/documentation/9.5/refman/py_env_start.html page], which discusses how to implement this same idea of using a single environment and thus a single license token with multiple models.  Other programs when run in parallel that call Gurobi such as R can also easily trigger the problem especially when many simultaneous jobs are submitted and/or run in parallel.
Note that all Gurobi license checkouts are handled by a single license server located in Ontario; it is therefore important to limit license checkout attempts as much as possible.  Rather than checking out a license for each invocation of Gurobi in a job---which may occur dozens or even hundreds of times---you should ensure that your program, whatever the language or computing environment used, only makes a single license checkout and then reuses this license token throughout the lifetime of the job. This will improve your job's performance because contacting a remote license server is very costly in time; moreover, responsiveness of our license server for everyone using Gurobi will also improve.  <span style="color:red">Failure to use Gurobi carefully in this regard may ultimately result in random intermittent license checkout failures for all users.  If this happens, you will be contacted and asked to kill all your jobs until your program is fixed and tested to ensure the problem is resolved.</span>   Some documentation on this subject for C++ programs may be found [https://www.gurobi.com/documentation/9.5/refman/cpp_env2.html here], explaining how to create a single Gurobi environment which can then be used for all your models. Python users can consult this [https://www.gurobi.com/documentation/9.5/refman/py_env_start.html page], which discusses how to implement this same idea of using a single environment and thus a single license token with multiple models.  Other programs that call Gurobi, such as R, can also easily trigger the problem when run in parallel, especially when many simultaneous parallel jobs are submitted and/or run.


== Interactive Allocations == <!--T:5-->
== Interactive allocations == <!--T:5-->


===Gurobi Command-Line Tools=== <!--T:51-->
===Gurobi command-line tools=== <!--T:51-->


  <!--T:42-->
  <!--T:42-->
Line 41: Line 41:
  [gra800:~] gurobi_cl --help
  [gra800:~] gurobi_cl --help


===Gurobi Interactive Shell === <!--T:53-->
===Gurobi interactive shell === <!--T:53-->
   
   
  [gra-login2:~] salloc --time=1:00:0 --cpus-per-task=8 --mem=1G --account=def-xyz
  [gra-login2:~] salloc --time=1:00:0 --cpus-per-task=8 --mem=1G --account=def-xyz
Line 74: Line 74:
Reference: https://www.gurobi.com/documentation/8.1/refman/recording_api_calls.html
Reference: https://www.gurobi.com/documentation/8.1/refman/recording_api_calls.html


== Cluster Batch Job Submission == <!--T:7-->
== Cluster batch job submission == <!--T:7-->


<!--T:70-->
<!--T:70-->
Once a Slurm script has been prepared for a Gurobi problem, it can be submitted to the queue by running the <code>sbatch script-name.sh</code> command.  The jobs status in the queue can then be checked by running the <code>sq</code> command.  The following Slurm scripts demonstrate solving 2 problems provided in the <tt> examples</tt> directory of each Gurobi module.
Once a Slurm script has been prepared for a Gurobi problem, it can be submitted to the queue by running the <code>sbatch script-name.sh</code> command.  The jobs status in the queue can then be checked by running the <code>sq</code> command.  The following Slurm scripts demonstrate solving 2 problems provided in the <tt> examples</tt> directory of each Gurobi module.


=== Data Example === <!--T:71-->  
=== Data example === <!--T:71-->  


<!--T:73-->
<!--T:73-->
Line 106: Line 106:
}}
}}


=== Python Example === <!--T:81-->  
=== Python example === <!--T:81-->  


<!--T:83-->
<!--T:83-->
This is an example Slurm script for solving a [https://www.gurobi.com/documentation/9.5/examples/a_list_of_the_grb_examples.html simple facility location model] with [https://www.gurobi.com/documentation/9.5/examples/facility_py.html Gurobi Python].  The example shows how to set the threads  [https://www.gurobi.com/documentation/9.5/refman/parameters.html#sec:Parameters parameter] with a dynamically created [https://www.gurobi.com/documentation/9.5/quickstart_linux/using_a_grb_env_file.html gurobi.env] file when using the [https://www.gurobi.com/documentation/9.5/refman/python_parameter_examples.html Gurobi python interface].  This must be done in the working directory for each submitted job, otherwise Gurobi will (by default) start as many execute [https://www.gurobi.com/documentation/9.5/refman/threads.html#parameter:Threads threads] as there are physical cores on the compute node, instead of using the number of physical cores allocated to the job by Slurm, potentially slowing down the job and negatively impacting other user jobs running on the same node.
This is an example Slurm script for solving a [https://www.gurobi.com/documentation/9.5/examples/a_list_of_the_grb_examples.html simple facility location model] with [https://www.gurobi.com/documentation/9.5/examples/facility_py.html Gurobi Python].  The example shows how to set the threads  [https://www.gurobi.com/documentation/9.5/refman/parameters.html#sec:Parameters parameter] equal to the number of cores allocated to a job by dynamically generating a [https://www.gurobi.com/documentation/9.5/quickstart_linux/using_a_grb_env_file.html gurobi.env] file into the working directory when using the [https://www.gurobi.com/documentation/9.5/refman/python_parameter_examples.html Gurobi python interface].  This must be done for each submitted job, otherwise Gurobi will (by default) start as many execute [https://www.gurobi.com/documentation/9.5/refman/threads.html#parameter:Threads threads] as there are physical cores on the compute node potentially slowing down the job and negatively impacting other user jobs running on the same node.
{{File
{{File
   |name=script-py_facility.sh
   |name=script-facility.sh
   |lang="sh"
   |lang="sh"
   |contents=
   |contents=
Line 122: Line 122:


<!--T:84-->
<!--T:84-->
#module load StdEnv/2016     # for versions < 9.0.3  
#module load StdEnv/2020     # for versions < 10.0.3
module load StdEnv/2020       # for versions > 9.0.2
module load StdEnv/2023       # for versions > 10.0.3


<!--T:85-->
<!--T:85-->
module load gurobi/9.5.0
module load gurobi/11.0.1


<!--T:86-->
<!--T:86-->
Line 138: Line 138:


<!--T:91-->
<!--T:91-->
Gurobi brings it's own version of Python which does not contain any 3rd-party Python packages except Gurobi.  In order to use Gurobi together with popular Python packages like NumPy, Matplotlib, Pandas and others, we need to create a [[Python#Creating_and_using_a_virtual_environment|virtual Python environment]] in which we can install both <code>gurobipy</code> and for example <code>pandas</code>.  Before we start, we need to decide which combination of versions for Gurobi and Python to use.  A listing of the Python version(s) supported by the major gurobi versions installed in the previous through current standard environments (StdEnv) follows:
Gurobi brings it's own version of Python which does not contain any 3rd-party Python packages except Gurobi.  In order to use Gurobi together with popular Python packages like NumPy, Matplotlib, Pandas and others, we need to create a [[Python#Creating_and_using_a_virtual_environment|virtual Python environment]] in which we can install both <code>gurobipy</code> and for example <code>pandas</code>.  Before we start, we need to decide which combination of versions for Gurobi and Python to use.  Following is a list of the Python versions supported by the major Gurobi versions installed in the previous through current standard environments (StdEnv):


<!--T:201-->
<!--T:201-->
Line 153: Line 153:
  python3.11
  python3.11


=== HowTo Install Gurobi for Python ===
=== Installing Gurobi for Python === <!--T:218-->


As explained at the bottom of [[https://support.gurobi.com/hc/en-us/articles/360044290292-How-do-I-install-Gurobi-for-Python|How-do-I-install-Gurobi-for-Python]] the previous recommended approach for installing gurobi for python with setup.py has been deprecated as of version Gurobi 11.  For continuity a modified version of the instruction (usable under StdEnv/2023) is provided in the following section for Gurobi 10 and older.  Following this a new section is provided showing how to install gurobi for python from a wheel for version Gurobi 11 and greater.
<!--T:219-->
As mentioned near the end of this official document [http://support.gurobi.com/hc/en-us/articles/360044290292-How-do-I-install-Gurobi-for-Python How-do-I-install-Gurobi-for-Python], the previously recommended method for installing Gurobi for Python with <code>setup.py</code> has been deprecated to only be usable with Gurobi 10 versions (and older).  Therefore, a new section has been added below which shows how to simultaneously download a compatible binary wheel from [https://pypi.org/project/gurobipy/ pypi.org] and convert it into a format usable with the newly recommended command to install Gurobi for Python with Gurobi 11 versions (and newer).


=== Gurobi Versions 10.0.3 (and older) === <!--T:10-->
=== Gurobi versions 10.0.3 (and older) === <!--T:10-->


<!--T:97-->
<!--T:97-->
The following steps need to be done once per system and are usable with StdEnv/2023 and older.  First load the modules to [[Python#Creating_and_using_a_virtual_environment|create the virtual environment]] and activate it:
The following steps need to be done once per system and are usable with StdEnv/2023 and older.  First, load the modules to [[Python#Creating_and_using_a_virtual_environment|create the virtual environment]] and activate it:


<!--T:105-->
<!--T:105-->
Line 168: Line 169:
| source ~/env_gurobi/bin/activate
| source ~/env_gurobi/bin/activate
}}
}}
Now install any Python packages we want to use (in this case <code>pandas</code>) for example:
Now install any Python packages you want to use, in this case <code>pandas</code>:


<!--T:110-->
<!--T:110-->
Line 174: Line 175:
|  pip install --no-index  pandas
|  pip install --no-index  pandas
}}
}}
Next install gurobipy into the environment.  Note that as of StdEnv/2023 the install can no longer be done under $EBROOTGUROBI as previously documented using the command <code>python setup.py build --build-base /tmp/${USER} install</code> since a fatal <code>error: could not create 'gurobipy.egg-info': Read-only file system</code> crash message will occur.  Instead the required files need to copied to elsewhere (such as /tmp/$USER) and installed from there, for example:
Next, install gurobipy in the environment.  Note that as of StdEnv/2023 the installation can no longer be done under $EBROOTGUROBI using the command <code>python setup.py build --build-base /tmp/${USER} install</code> since a fatal error (<code>error: could not create 'gurobipy.egg-info': Read-only file system</code>) will occur.  Instead, the required files need to be copied elsewhere (such as /tmp/$USER) and the installation made from there, for example:


<!--T:113-->
<!--T:113-->
Line 186: Line 187:
!!
!!


         ********************************************************************************
         <!--T:220-->
********************************************************************************
         newlines are not allowed in `summary` and will break in the future
         newlines are not allowed in `summary` and will break in the future
         ********************************************************************************
         ********************************************************************************


<!--T:221-->
!!
!!
   write_field('Summary', single_line(summary))
   write_field('Summary', single_line(summary))
removing /tmp/roberpj/build
removing /tmp/roberpj/build
(env_gurobi) [roberpj@gra-login1:/tmp/roberpj]
(env_gurobi) [roberpj@gra-login1:/tmp/roberpj]
| deactivate
[name@server ~]
}}
}}
=== Gurobi Versions 11.0.0 (and newer) === <!--T:11-->


<!--T:97-->
=== Gurobi versions 11.0.0 (and newer) === <!--T:11-->
Once again, following steps need to be done once per system and are usable with StdEnv/2023 and older.  First load the modules to [[Python#Creating_and_using_a_virtual_environment|create the virtual environment]] and activate it.  Version 11.0.0 is skipped since it has been observed to seg fault in at least one example versus Version 11.0.1 which runs smoothly.
 
<!--T:98-->
Once again, the following steps need to be done once per system and are usable with StdEnv/2023 and older.  First load the modules to [[Python#Creating_and_using_a_virtual_environment|create the virtual environment]] and activate it.  Version 11.0.0 is skipped since it has been observed to seg fault in at least one example versus Version 11.0.1 which runs smoothly.


<!--T:105-->
<!--T:106-->
{{Commands|prompt=[name@server ~] $
{{Commands|prompt=[name@server ~] $
| module load gurobi/11.0.1 python
| module load gurobi/11.0.1 python
Line 206: Line 212:
| source ~/env_gurobi/bin/activate
| source ~/env_gurobi/bin/activate
}}
}}
As before, install any needed Python packages (in this case <code>pandas</code>) for example:
As before, install any needed Python packages.  Since the following matrix example requires <code>numpy</code>, we install the pandas package:


<!--T:110-->
<!--T:111-->
{{Commands|prompt=(env_gurobi) [name@server ~] $
{{Commands|prompt=(env_gurobi) [name@server ~] $
|  pip install --no-index  pandas
|  pip install --no-index  pandas
}}
}}
Next install gurobipy into the environment. As mentioned above and in [[https://support.gurobi.com/hc/en-us/articles/360044290292-How-do-I-install-Gurobi-for-Python|this article]] the use of setup.py to install Gurobi for python is deprecated starting with Gurobi 11.  Both pip and conda are given as alternatives, however since conda should not be used on alliance systems, the pip approach will be demonstrated here.
Next install gurobipy into the environment. As mentioned above and in [[https://support.gurobi.com/hc/en-us/articles/360044290292-How-do-I-install-Gurobi-for-Python|this article]] the use of setup.py to install Gurobi for python is deprecated starting with Gurobi 11.  Both pip and conda are given as alternatives; however, since conda should not be used on alliance systems, the pip approach will be demonstrated here. The installation of gurobipy is slightly complicated since our Linux systems are set up with gentoo prefix.  As a result neither A) the recommended command to download and install the gurobipy extension from the public PyPI server <code>pip install gurobipy==11.0.1</code> mentioned in the article line or B) the offline command to install the wheel with <code>python -m pip install --find-links <wheel-dir> --no-index gurobipy</code>, will work.  Instead a script available from the alliance maybe utilized to download and simultaneously convert the existing wheel into a usable format with a new name.  There is one caveat, for each new Gurobi version the researcher must go into https://pypi.org/project/gurobipy/11.0.1/#history and click on the desired version followed by the <code>Download files</code> button located in the menu on the left.  Finally, click to copy the https link for the wheel file (named gurobipy-11.0.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl in the case of Gurobi 11.0.1) and paste it as the --url argument as shown below :
 
The installation is slightly complicated since Alliance linux systems are setup with gentoo prefix.  As a result neither A) the recommended command to download and install the gurobipy extension from the public PyPI server <code>pip install gurobipy==11.0.1</code> mentioned in the article line or B) the offline command to install the wheel with <code>python -m pip install --find-links <wheel-dir> --no-index gurobipy</code>, will work.  Instead a script available from the alliance maybe utilized to download and modify the existing wheel into an usable format under a new name.  There is one caveat which is for each new Gurobi version the researcher must go into https://pypi.org/project/gurobipy/11.0.1/#history, click on the desired version followed by the <code>Download files</code> button in the left hand menu and finally copy the https link for the wheel file named gurobipy-11.0.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl in the case of Gurobi 11.0.1 and paste its link as the --url argument as shown in the following:


<!--T:112-->
<!--T:112-->
Line 221: Line 225:
| chmod u+rx unmanylinuxize.sh
| chmod u+rx unmanylinuxize.sh
| ./unmanylinuxize.sh --package gurobipy --version 11.0.1 --url  
| ./unmanylinuxize.sh --package gurobipy --version 11.0.1 --url  
https://files.pythonhosted.org/packages/c5/c6/7cb959ea454c58169ecc6d165c8e3e915769d963df1bab61f5c5bc0052e6/
https://files.pythonhosted.org/packages/1c/96/4c800e7cda4a1688d101a279087646912cf432b0f61ff5c816f0bc8503e0/gurobipy-11.0.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
gurobipy-11.0.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| ls
'gurobipy-11.0.1-cp312-cp312-linux_x86_64.whl' -> '../gurobipy-11.0.1-cp312-cp312-linux_x86_64.whl'
gurobipy-11.0.1-cp311-cp311-linux_x86_64.whl unmanylinuxize.sh
| python -m pip install --find-links ~/pythonwheels --no-index gurobipy}}
| python -m pip install --find-links $PWD --no-index gurobipy
| deactivate
[name@server ~]}}
 
=== Running Gurobi in the environment === <!--T:120-->
 
<!--T:126-->
Once created our Gurobi environment can be activated and used at any time.  To demonstrate this we also load gurobi (so $EBROOTGUROBI is defined) and <code>scipy-stack</code> (so scipy is available).  Both are required to run the matrix example (along with numpy that was already installed into our  environment with pip in a previous step above via pandas).
 
<!--T:203-->
{{Commands|prompt=[name@server ~] $
| module load gurobi/11.0.1 scipy-stack
  source ~/env_gurobi/bin/activate
  (env_gurobi) [name@server ~]
}}


=== Using the virtual environment === <!--T:120-->  
<!--T:205-->
Python scripts, such as the examples provided with the gurobi module can now be run (within the virtual environment) using python :
 
<!--T:222-->
{{Commands|prompt=(env_gurobi) [name@server ~] $
| python $EBROOTGUROBI/examples/python/matrix1.py
}}


<!--T:122-->
<!--T:122-->
Python scripts can now import both Pandas and Gurobi:
Likewise custom python scripts such as the following can be run as jobs in the queue by writing slurm scripts that load your virtual environment.


<!--T:202-->
<!--T:223-->
{{Commands|prompt=[name@server ~] $
{{Commands|prompt=[name@server ~] $
| cat my_gurobi_script.py
| cat my_gurobi_script.py
Line 239: Line 263:
   from gurobipy import *
   from gurobipy import *
   etc
   etc
}}
<!--T:126-->
Once created we can activate our Gurobi environment at any time with
<!--T:203-->
{{Commands|prompt=[name@server ~] $
| module load gurobi/11.0.1
  source ~/env_gurobi/bin/activate
  (env_gurobi) python my_gurobi_script.py
}}
}}


<!--T:130-->
<!--T:224-->
Note that we now use <code>python</code> instead of <code>gurobi.sh</code>!  This is an example job script that we can use:
Submit your script to the queue by running <code>sbatch my_slurm_script.sh</code> as per usual :


<!--T:204-->
<!--T:204-->
{{File
{{File
   |name=gurobi-py_example.sh
   |name=my_slurm_script.sh
   |lang="sh"
   |lang="sh"
   |contents=
   |contents=
#!/bin/bash
#!/bin/bash
#SBATCH --time=0-00:30       # time limit (D-HH:MM)
#SBATCH --account=def-somegrp  # specify an account
#SBATCH --cpus-per-task=4     # number of CPUs to use
#SBATCH --time=0-00:30         # time limit (D-HH:MM)
#SBATCH --mem-per-cpu=1000M  # memory per CPU (in MB)
#SBATCH --nodes=1              # run job on one node
#SBATCH --cpus-per-task=4     # specify number of CPUS
#SBATCH --mem=4000M            # specify total MB memory


<!--T:205-->
<!--T:225-->
module load StdEnv/2023
module load StdEnv/2023
module load gurobi/11.0.1
module load gurobi/11.0.1
# module load scipy-stack      # uncomment if needed


<!--T:206-->
<!--T:226-->
echo "Threads ${SLURM_CPUS_ON_NODE:-1}" > gurobi.env
 
<!--T:227-->
source ~/env_gurobi/bin/activate
source ~/env_gurobi/bin/activate
python my_gurobi_script.py
}}


<!--T:207-->
<!--T:206-->
echo "Threads ${SLURM_CPUS_ON_NODE:-1}" > gurobi.env  # set number of threads
Further information regarding how to create and use python virtual environments within job scripts can be found [[Python#Creating_virtual_environments_inside_of_your_jobs|here]].
 
<!--T:208-->
python  my_gurobi_script.py
}}


== Using Gurobi with Java == <!--T:210-->
== Using Gurobi with Java == <!--T:210-->

Latest revision as of 12:55, 22 July 2024

Other languages:

Gurobi is a commercial software suite for solving complex optimization problems. This wiki page describes the non-commercial use of Gurobi software on our clusters.

License limitations

We support and provide a free license to use Gurobi on the Graham, Cedar, Béluga and Niagara clusters. The license provides a total number of 4096 simultaneous uses (tokens in use) and permits distributed optimization with up to 100 nodes. A single user can run multiple simultaneous jobs. In order to use Gurobi, you must agree to certain conditions. Please contact support and include a copy of the following completed agreement. You will then be added into our license file as a user within a few days.

Academic usage agreement

My Alliance username is "_______" and I am a member of the academic institution "_____________________." This message confirms that I will only use the Gurobi license provided on Digital Research Alliance of Canada systems for the purpose of non-commercial research project(s) to be published in publicly available article(s).

Configuring your account

You do NOT need to create a ~/.licenses/gurobi.lic file. The required settings to use our Gurobi license are configured by default when you load a Gurobi module on any cluster. To verify your username has been added to our Gurobi license and is working properly, run the following command: 

$ module load gurobi
$ gurobi_cl 1> /dev/null && echo Success || echo Fail

If it returns "Success" you can begin using Gurobi immediately. If the test returns "Fail" then check whether a file named ~/.license/gurobi exists; if so, rename or remove this file, reload the module and try the test again. If it still returns "Fail", check whether there are any environment variables containing GUROBI defined in either of our your ~/.bashrc or ~/.bash_profile files. If you find any, comment or remove those lines, log out and log in again, reload the Gurobi module and try the test again. If you still get "Fail", contact support for help.

Minimizing license checkouts

Note that all Gurobi license checkouts are handled by a single license server located in Ontario; it is therefore important to limit license checkout attempts as much as possible. Rather than checking out a license for each invocation of Gurobi in a job---which may occur dozens or even hundreds of times---you should ensure that your program, whatever the language or computing environment used, only makes a single license checkout and then reuses this license token throughout the lifetime of the job. This will improve your job's performance because contacting a remote license server is very costly in time; moreover, responsiveness of our license server for everyone using Gurobi will also improve. Failure to use Gurobi carefully in this regard may ultimately result in random intermittent license checkout failures for all users. If this happens, you will be contacted and asked to kill all your jobs until your program is fixed and tested to ensure the problem is resolved. Some documentation on this subject for C++ programs may be found here, explaining how to create a single Gurobi environment which can then be used for all your models. Python users can consult this page, which discusses how to implement this same idea of using a single environment and thus a single license token with multiple models. Other programs that call Gurobi, such as R, can also easily trigger the problem when run in parallel, especially when many simultaneous parallel jobs are submitted and/or run.

Interactive allocations

Gurobi command-line tools

[gra-login2:~] salloc --time=1:00:0 --cpus-per-task=8 --mem=1G --account=def-xyz
[gra800:~] module load gurobi
[gra800:~] gurobi_cl Record=1 Threads=8 Method=2 ResultFile=p0033.sol LogFile=p0033.log $GUROBI_HOME/examples/data/p0033.mps
[gra800:~] gurobi_cl --help

Gurobi interactive shell

[gra-login2:~] salloc --time=1:00:0 --cpus-per-task=8 --mem=1G --account=def-xyz
[gra800:~] module load gurobi
[gra800:~] echo "Record 1" > gurobi.env    see *
[gra800:~] gurobi.sh
gurobi> m = read('/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/gurobi/8.1.1/examples/data/glass4.mps') 
gurobi> m.Params.Threads = 8               see **
gurobi> m.Params.Method = 2
gurobi> m.Params.ResultFile = "glass4.sol"
gurobi> m.Params.LogFile = "glass4.log"
gurobi> m.optimize()
gurobi> m.write('glass4.lp')
gurobi> m.status                           see ***
gurobi> m.runtime                          see ****
gurobi> help()

where 

   * https://www.gurobi.com/documentation/8.1/refman/recording_api_calls.html
  ** https://www.gurobi.com/documentation/8.1/refman/parameter_descriptions.html
 *** https://www.gurobi.com/documentation/8.1/refman/optimization_status_codes.html
**** https://www.gurobi.com/documentation/8.1/refman/attributes.html

Replaying API calls

You can record API calls and repeat them with 

[gra800:~] gurobi_cl recording000.grbr

Reference: https://www.gurobi.com/documentation/8.1/refman/recording_api_calls.html

Cluster batch job submission

Once a Slurm script has been prepared for a Gurobi problem, it can be submitted to the queue by running the sbatch script-name.sh command. The jobs status in the queue can then be checked by running the sq command. The following Slurm scripts demonstrate solving 2 problems provided in the examples directory of each Gurobi module.

Data example

The following Slurm script utilizes the Gurobi command-line interface to solve a simple coin production model written in LP format. The last line demonstrates how parameters can be passed directly to the Gurobi command-line tool gurobi_cl using simple command line arguments. For help selecting which parameters are best used for a particular problem and for choosing optimal values, refer to both the Performance and Parameters and Algorithms and Search sections found in the Gurobi Knowledge Base as well as the extensive online Gurobi documentation.

File : script-lp_coins.sh

#!/bin/bash
#SBATCH --account=def-group   # some account
#SBATCH --time=0-00:30        # specify time limit (D-HH:MM)
#SBATCH --cpus-per-task=8     # specify number threads
#SBATCH --mem=4G              # specify total memory
#SBATCH --nodes=1             # do not change

#module load StdEnv/2016      # for versions < 9.0.3 
module load StdEnv/2020       # for versions > 9.0.2

module load gurobi/9.5.0

rm -f coins.sol
gurobi_cl Threads=$SLURM_CPUS_ON_NODE Method=2 ResultFile=coins.sol ${GUROBI_HOME}/examples/data/coins.lp


Python example

This is an example Slurm script for solving a simple facility location model with Gurobi Python. The example shows how to set the threads parameter equal to the number of cores allocated to a job by dynamically generating a gurobi.env file into the working directory when using the Gurobi python interface. This must be done for each submitted job, otherwise Gurobi will (by default) start as many execute threads as there are physical cores on the compute node potentially slowing down the job and negatively impacting other user jobs running on the same node.

File : script-facility.sh

#!/bin/bash
#SBATCH --account=def-group   # some account
#SBATCH --time=0-00:30        # specify time limit (D-HH:MM)
#SBATCH --cpus-per-task=4     # specify number threads
#SBATCH --mem=4G              # specify total memory
#SBATCH --nodes=1             # do not change

#module load StdEnv/2020      # for versions < 10.0.3
module load StdEnv/2023       # for versions > 10.0.3

module load gurobi/11.0.1

echo "Threads ${SLURM_CPUS_ON_NODE:-1}" > gurobi.env

gurobi.sh ${GUROBI_HOME}/examples/python/facility.py


Using Gurobi in Python virtual environments

Gurobi brings it's own version of Python which does not contain any 3rd-party Python packages except Gurobi. In order to use Gurobi together with popular Python packages like NumPy, Matplotlib, Pandas and others, we need to create a virtual Python environment in which we can install both gurobipy and for example pandas. Before we start, we need to decide which combination of versions for Gurobi and Python to use. Following is a list of the Python versions supported by the major Gurobi versions installed in the previous through current standard environments (StdEnv): 

[name@server ~] module load StdEnv/2016; module load gurobi/8.1.1; cd $EBROOTGUROBI/lib; ls -d python*
python2.7  python2.7_utf16  python2.7_utf32  python3.5_utf32  python3.6_utf32  python3.7_utf32
 
[name@server ~] module load StdEnv/2020; module load gurobi/9.5.2; cd $EBROOTGUROBI/lib; ls -d python*
python2.7_utf16  python2.7_utf32  python3.10_utf32  python3.7  python3.7_utf32  python3.8_utf32  python3.9_utf32

[name@server ~] module load StdEnv/2023; module load gurobi/10.0.3; cd $EBROOTGUROBI/lib; ls -d python*
python3.10_utf32  python3.11_utf32  python3.7  python3.7_utf32  python3.8_utf32  python3.9_utf32

[name@server ~] module load StdEnv/2023; module load gurobi/11.0.1; cd $EBROOTGUROBI/lib; ls -d python*
python3.11

Installing Gurobi for Python

As mentioned near the end of this official document How-do-I-install-Gurobi-for-Python, the previously recommended method for installing Gurobi for Python with setup.py has been deprecated to only be usable with Gurobi 10 versions (and older). Therefore, a new section has been added below which shows how to simultaneously download a compatible binary wheel from pypi.org and convert it into a format usable with the newly recommended command to install Gurobi for Python with Gurobi 11 versions (and newer).

Gurobi versions 10.0.3 (and older)

The following steps need to be done once per system and are usable with StdEnv/2023 and older. First, load the modules to create the virtual environment and activate it: 

[name@server ~] $ module load gurobi/10.0.3 python
[name@server ~] $ virtualenv --no-download  ~/env_gurobi
[name@server ~] $ source ~/env_gurobi/bin/activate

Now install any Python packages you want to use, in this case pandas: 

(env_gurobi) [name@server ~] $ pip install --no-index  pandas

Next, install gurobipy in the environment. Note that as of StdEnv/2023 the installation can no longer be done under $EBROOTGUROBI using the command python setup.py build --build-base /tmp/${USER} install since a fatal error (error: could not create 'gurobipy.egg-info': Read-only file system) will occur. Instead, the required files need to be copied elsewhere (such as /tmp/$USER) and the installation made from there, for example: 

(env_gurobi) [name@server ~] $ mkdir /tmp/$USER
(env_gurobi) [name@server ~] $ cp -r $EBROOTGUROBI/{lib,setup.py} /tmp/$USER
(env_gurobi) [name@server ~] $ cd /tmp/$USER
(env_gurobi) [name@server ~] $ python setup.py install
(env_gurobi) [roberpj@gra-login1:/tmp/roberpj] python setup.py install
/home/roberpj/env_gurobi/lib/python3.11/site-packages/setuptools/_core_metadata.py:158: SetuptoolsDeprecationWarning: Invalid config.
!!

        ********************************************************************************
        newlines are not allowed in `summary` and will break in the future
        ********************************************************************************

!!
  write_field('Summary', single_line(summary))
removing /tmp/roberpj/build
(env_gurobi) [roberpj@gra-login1:/tmp/roberpj]
(env_gurobi) [name@server ~] $ deactivate
[name@server ~]


Gurobi versions 11.0.0 (and newer)

Once again, the following steps need to be done once per system and are usable with StdEnv/2023 and older. First load the modules to create the virtual environment and activate it. Version 11.0.0 is skipped since it has been observed to seg fault in at least one example versus Version 11.0.1 which runs smoothly. 

[name@server ~] $ module load gurobi/11.0.1 python
[name@server ~] $ virtualenv --no-download  ~/env_gurobi
[name@server ~] $ source ~/env_gurobi/bin/activate

As before, install any needed Python packages. Since the following matrix example requires numpy, we install the pandas package: 

(env_gurobi) [name@server ~] $ pip install --no-index  pandas

Next install gurobipy into the environment. As mentioned above and in [article] the use of setup.py to install Gurobi for python is deprecated starting with Gurobi 11. Both pip and conda are given as alternatives; however, since conda should not be used on alliance systems, the pip approach will be demonstrated here. The installation of gurobipy is slightly complicated since our Linux systems are set up with gentoo prefix. As a result neither A) the recommended command to download and install the gurobipy extension from the public PyPI server pip install gurobipy==11.0.1 mentioned in the article line or B) the offline command to install the wheel with python -m pip install --find-links <wheel-dir> --no-index gurobipy, will work. Instead a script available from the alliance maybe utilized to download and simultaneously convert the existing wheel into a usable format with a new name. There is one caveat, for each new Gurobi version the researcher must go into https://pypi.org/project/gurobipy/11.0.1/#history and click on the desired version followed by the Download files button located in the menu on the left. Finally, click to copy the https link for the wheel file (named gurobipy-11.0.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl in the case of Gurobi 11.0.1) and paste it as the --url argument as shown below : 

(env_gurobi) [name@server ~] $ wget https://raw.githubusercontent.com/ComputeCanada/wheels_builder/main/unmanylinuxize.sh
(env_gurobi) [name@server ~] $ chmod u+rx unmanylinuxize.sh
(env_gurobi) [name@server ~] $ ./unmanylinuxize.sh --package gurobipy --version 11.0.1 --url 
https://files.pythonhosted.org/packages/1c/96/4c800e7cda4a1688d101a279087646912cf432b0f61ff5c816f0bc8503e0/gurobipy-11.0.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
(env_gurobi) [name@server ~] $ ls
gurobipy-11.0.1-cp311-cp311-linux_x86_64.whl  unmanylinuxize.sh
(env_gurobi) [name@server ~] $ python -m pip install --find-links $PWD --no-index gurobipy
(env_gurobi) [name@server ~] $ deactivate
[name@server ~]


Running Gurobi in the environment

Once created our Gurobi environment can be activated and used at any time. To demonstrate this we also load gurobi (so $EBROOTGUROBI is defined) and scipy-stack (so scipy is available). Both are required to run the matrix example (along with numpy that was already installed into our environment with pip in a previous step above via pandas). 

[name@server ~] $ module load gurobi/11.0.1 scipy-stack
  source ~/env_gurobi/bin/activate
  (env_gurobi) [name@server ~]


Python scripts, such as the examples provided with the gurobi module can now be run (within the virtual environment) using python : 

(env_gurobi) [name@server ~] $ python $EBROOTGUROBI/examples/python/matrix1.py


Likewise custom python scripts such as the following can be run as jobs in the queue by writing slurm scripts that load your virtual environment. 

[name@server ~] $ cat my_gurobi_script.py
  import pandas as pd
  import numpy as np
  import gurobipy as gurobi
  from gurobipy import *
  etc


Submit your script to the queue by running sbatch my_slurm_script.sh as per usual :


File : my_slurm_script.sh

#!/bin/bash
#SBATCH --account=def-somegrp  # specify an account
#SBATCH --time=0-00:30         # time limit (D-HH:MM)
#SBATCH --nodes=1              # run job on one node
#SBATCH --cpus-per-task=4      # specify number of CPUS
#SBATCH --mem=4000M            # specify total MB memory

module load StdEnv/2023
module load gurobi/11.0.1
# module load scipy-stack      # uncomment if needed

echo "Threads ${SLURM_CPUS_ON_NODE:-1}" > gurobi.env

source ~/env_gurobi/bin/activate
python my_gurobi_script.py


Further information regarding how to create and use python virtual environments within job scripts can be found here.

Using Gurobi with Java

To use Gurobi with Java, you will also need to load a Java module and add an option to your Java command in order to allow the Java virtual environment to find the Gurobi libraries. A sample job script is below:

File : gurobi-java.sh

#!/bin/bash
#SBATCH --time=0-00:30        # time limit (D-HH:MM)
#SBATCH --cpus-per-task=1     # number of CPUs (threads) to use
#SBATCH --mem=4096M           # memory per CPU (in MB)

module load java/14.0.2
module load gurobi/9.1.2

java -Djava.library.path=$EBROOTGUROBI/lib -Xmx4g -jar my_java_file.jar


Using Gurobi with Jupyter notebooks

Various topics can be found by visiting Resources, then clicking Code and Modeling Examples and finally Optimization with Python – Jupyter Notebook Modeling Examples. Alternatively visit support.gurobi.com and search on Jupyter Notebooks.

A demo case of using Gurobi with Jupyter notebooks on our systems can be found in this video recording, i.e. at time 38:28.

Cite Gurobi

Please see How do I cite Gurobi software for an academic publication?

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=Gurobi&oldid=158008"