Advanced Jupyter configuration

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

Introduction

• Project Jupyter: "a non-profit, open-source project, born out of the IPython Project in 2014 as it evolved to support interactive data science and scientific computing across all programming languages."
• JupyterLab: "a web-based interactive development environment for notebooks, code, and data. Its flexible interface allows users to configure and arrange workflows in data science, scientific computing, computational journalism, and machine learning. A modular design allows for extensions that expand and enrich functionality."

A JupyterLab server should only run on a compute node or on a cloud instance; cluster login nodes are not a good choice because they impose various limits which can stop applications if they consume too much CPU time or memory. In the case of using a compute node, users can reserve compute resources by submitting a job that requests a specific number of CPUs (and optionally GPUs), an amount of memory and the run time. In this page, we give detailed instructions on how to configure and submit a JupyterLab job on any national cluster.

But, what about ...

• Jupyter Notebook? JupyterLab is a more modern and flexible interface than the classic Jupyter Notebook
• A preconfigured JupyterLab service? Some regional partners provide a web portal named JupyterHub. For instance, many of these portals offer a preconfigured JupyterLab service so that users do not have to create their own setup. To learn more, visit the JupyterHub wiki page

Installing JupyterLab

These instructions install JupyterLab with the pip command in a Python virtual environment:

1. Load a Python module, either the default one (as shown below) or a specific version (see available versions with module avail python):

   [name@server ~]$ module load python
   

2. Create a new Python virtual environment:

   [name@server ~]$ virtualenv --no-download $HOME/jupyter_py3
   

3. Activate your newly created Python virtual environment:

   [name@server ~]$ source $HOME/jupyter_py3/bin/activate
   

4. Install JupyterLab in your new virtual environment (note: it takes a few minutes):

   (jupyter_py3) [name@server ~]$ pip install --no-index jupyterlab
   

5. In the virtual environment, create a wrapper script that launches JupyterLab:

   (jupyter_py3) [name@server ~]$ echo -e '#!/bin/bash\nunset XDG_RUNTIME_DIR\njupyter-lab --ip $(hostname -f) --no-browser' > $VIRTUAL_ENV/bin/jupyterlab.sh
   

6. Finally, make the script executable:

   (jupyter_py3) [name@server ~]$ chmod u+x $VIRTUAL_ENV/bin/jupyterlab.sh
   

Installing extensions

Extensions allow you to add functionalities and modify the JupyterLab’s user interface.

Jupyter Lmod

Jupyter Lmod is an extension that allows you to interact with environment modules before launching kernels. The extension uses the Lmod's Python interface to accomplish module-related tasks like loading, unloading, saving a collection, etc.

The following commands will install and enable the Jupyter Lmod extension in your environment (note: the third command takes a few minutes to complete):

(jupyter_py3) [name@server ~]$ module load nodejs
(jupyter_py3) [name@server ~]$ pip install jupyterlmod
(jupyter_py3) [name@server ~]$ jupyter labextension install jupyterlab-lmod

Instructions on how to configure software modules in the JupyterLab interface are provided in the JupyterHub page.

Using your installation

Activating the environment

Make sure the Python virtual environment in which you have installed JupyterLab is activated. For example, when you log into the cluster, you have to activate it again with:

[name@server ~]$ source $HOME/jupyter_py3/bin/activate

To verify that your environment is ready, you can get a list of installed jupyter* packages with the following command:

(jupyter_py3) [name@server ~]$ pip freeze | grep jupyter
jupyter-client==7.1.0+computecanada
jupyter-core==4.9.1+computecanada
jupyter-server==1.9.0+computecanada
jupyterlab==3.1.7+computecanada
jupyterlab-pygments==0.1.2+computecanada
jupyterlab-server==2.3.0+computecanada

Starting JupyterLab

To start a JupyterLab server, submit an interactive job with salloc. Adjust the parameters based on your needs. See Running jobs for more information.

(jupyter_py3) [name@server ~]$ salloc --time=1:0:0 --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1024M --account=def-yourpi srun $VIRTUAL_ENV/bin/jupyterlab.sh
...
[I 2021-12-06 10:37:14.262 ServerApp] jupyterlab | extension was successfully linked.
...
[I 2021-12-06 10:37:39.259 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2021-12-06 10:37:39.356 ServerApp]

    To access the server, open this file in a browser:
        file:///home/name/.local/share/jupyter/runtime/jpserver-198146-open.html
    Or copy and paste one of these URLs:
        http://node_name.int.cluster.computecanada.ca:8888/lab?token=101c3688298e78ab554ef86d93a196deaf5bcd2728fad4eb
     or http://127.0.0.1:8888/lab?token=101c3688298e78ab554ef86d93a196deaf5bcd2728fad4eb

Connecting to JupyterLab

To access JupyterLab running on a compute node from your web browser, you will need to create an SSH tunnel from your computer through the cluster since the compute nodes are not directly accessible from the Internet.

From Linux or macOS

On a Linux or macOS system, we recommend using the sshuttle Python package.

On your computer, open a new terminal window and create the SSH tunnel with the following sshuttle command where <username> must be substituted by your Compute Canada username, and <cluster> by the cluster on which you have launched JupyterLab:

[name@local ~]$ sshuttle --dns -Nr <username>@<cluster>.computecanada.ca

Then, copy and paste the first provided HTTP address into your Web browser. In the above salloc example, this would be:

http://node_name.int.cluster.computecanada.ca:8888/lab?token=101c3688298e78ab554ef86d93a196deaf5bcd2728fad4eb

From Windows

An SSH tunnel can be created from Windows using MobaXTerm as follows. Note: this procedure also works from a terminal in any Unix system (like Linux, macOS, etc).

1. Once JupyterLab is launched on a compute node (see Starting JupyterLab), you can extract the hostname:port and the token from the first provided HTTP address. For example:

   http://node_name.int.cluster.computecanada.ca:8888/lab?token=101c368829...2728fad4eb
          └────────────────────┬────────────────────┘           └──────────┬──────────┘
                         hostname:port                                   token
   

2. Open a new Terminal tab in MobaXTerm. In the following command, substitute <hostname:port> by its corresponding value (refer to the above figure), substitute <username> by your Compute Canada username, and substitute <cluster> by the cluster on which you have launched JupyterLab:

   [name@local ~]$ ssh -L 8888:<hostname:port> <username>@<cluster>.computecanada.ca
   

3. Open your Web browser and go to the following address where <token> must be substituted by the alphanumerical value extracted from the above figure:

   http://localhost:8888/?token=<token>
   

Shutting down JupyterLab

You can shut down the JupyterLab server before the walltime limit by pressing Ctrl-C twice in the terminal that launched the interactive job.

If you have used MobaXterm to create an SSH tunnel, press Ctrl-D to shut down the tunnel.

Adding kernels

It is possible to add kernels for other programming languages, for a different Python version or for a persistent virtual environment that has all required packages and libraries for your project. Refer to Making kernels for Jupyter to learn more.

The installation of a new kernel is done in two steps:

1. Installation of the packages that will allow the language interpreter to communicate with the Jupyter interface.
2. Creation of a file that will indicate to JupyterLab how to initiate a communication channel with the language interpreter. This file is called a kernel spec file, and it will be saved in a sub-folder of ~/.local/share/jupyter/kernels.

In the following sections, we provide a few examples of the kernel installation procedure.

Julia Kernel

Prerequisites:

1. The configuration of a Julia kernel depends on a Python virtual environment that already has all the Python Kernel dependencies. If you do not have such virtual environment, make sure to follow instructions listed in the next section.
2. Since the installation of Julia packages requires an access to Internet, the configuration of a Julia kernel must be done in a remote shell session on a login node.

Once you have a Python virtual environment available and activated, you may configure the Julia kernel:

1. Load the Julia module:

   (jupyter_py3) [name@server ~]$ module load julia
   

2. Install IJulia:

   (jupyter_py3) [name@server ~]$ echo -e 'using Pkg\nPkg.add("IJulia")' | julia
   

3. Important: start or restart a new JupyterLab session before using the Julia kernel.

For more information, see the IJulia documentation.

Installing more Julia packages

As in the above installation procedure, it is required to install Julia packages from a login node, but the Python virtual environment could be deactivated:

1. Make sure the same Julia module is loaded:

   [name@server ~]$ module load julia
   

2. Install any required package. For example with Glob:

   [name@server ~]$ echo -e 'using Pkg\nPkg.add("Glob")' | julia
   

3. The newly installed Julia packages should already be usable in a notebook executed by the Julia kernel.

Python Kernel

In a terminal with an active session on the remote server, you may configure a Python virtual environment with all the required Python modules and a custom Python kernel for JupyterLab. Here are the initial steps for the simplest Jupyter configuration in a new Python virtual environment:

1. Start from a clean Bash environment (this is required if you are using the Jupyter Terminal):

   [name@server ~]$ env -i HOME=$HOME bash -l
   

2. Load a Python module:

   [name@server ~]$ module load python/3.8
   

3. Create a new Python virtual environment:

   [name@server ~]$ virtualenv --no-download $HOME/jupyter_py3.8
   

4. Activate your newly created Python virtual environment:

   [name@server ~]$ source $HOME/jupyter_py3.8/bin/activate
   

5. Install the ipykernel library:

   (jupyter_py3.8) [name@server ~]$ pip install --no-index ipykernel
   

6. Create the common kernels folder:

   (jupyter_py3.8) [name@server ~]$ mkdir -p ~/.local/share/jupyter/kernels
   

7. Generate the kernel spec file. Substitute <unique_name> by a name that will uniquely identify your kernel:

   (jupyter_py3.8) [name@server ~]$ python -m ipykernel install --user --name <unique_name> --display-name "Python 3.8 Kernel"
   

8. Deactivate the virtual environment and exit the temporary Bash session:

   (jupyter_py3.8) [name@server ~]$ deactivate && exit
   

9. Start or restart a new JupyterLab session.

For more information, see the ipykernel documentation.

Installing more Python libraries

Based on the Python virtual environment created in the previous section:

1. Start from a clean Bash environment (this is required if you are using the Jupyter Terminal):

   [name@server ~]$ env -i HOME=$HOME bash -l
   

2. Activate the virtual environment:

   [name@server ~]$ source $HOME/jupyter_py3.8/bin/activate
   

3. Install any required library (for example: numpy):

   (jupyter_py3.8) [name@server ~]$ pip install --no-index numpy
   

4. Deactivate the virtual environment and exit the temporary Bash session:

   (jupyter_py3.8) [name@server ~]$ deactivate && exit
   

5. The newly installed Python libraries can now be imported in any notebook using the Python 3.8 Kernel.

R Kernel

Prerequisites:

1. Configuring an R kernel still depends on a Python virtual environment that already has all the Python Kernel dependencies. If you do not have such virtual environment, make sure to follow instructions listed in the previous section.
2. Since the installation of R packages requires an access to CRAN, the configuration of an R kernel must be done in a remote shell session on a login node. For example, from a local terminal or any SSH client:

   [my_computer ~] $ ssh username@cluster_address
   

Once you have a Python virtual environment available, you may configure the R kernel:

1. Activate the Python virtual environment.

   [name@server ~]$ source $HOME/jupyter_py3.8/bin/activate
   

2. Load an R module.

   (jupyter_py3.8) [name@server ~]$ module load r/4.1
   

3. Install the R kernel dependencies - this will take up to 10 minutes, and packages should be installed in a local directory like ~/R/x86_64-pc-linux-gnu-library/4.1.

   (jupyter_py3.8) [name@server ~]$ R --no-save
   

   > install.packages(c('crayon', 'pbdZMQ', 'devtools'), repos='http://cran.us.r-project.org')
   

4. Install the R kernel.

   > devtools::install_github(paste0('IRkernel/', c('repr', 'IRdisplay', 'IRkernel')))
   

5. Install the R kernel spec file.

   > IRkernel::installspec()
   

6. Quit R (with q()), deactivate the virtual environment and close the remote session:

   (jupyter_py3.8) [name@server ~]$ deactivate && exit
   

7. Start or restart a new JupyterLab session.

For more information, see the IRKernel documentation.

Installing more R packages

The installation of R packages cannot be done from Notebooks because there is no access to CRAN. As in the above installation procedure, it is required to install R packages from a login node:

1. From a local terminal or any SSH client, connect to a login node of the cluster hosting JupyterHub.
2. Load the same R module.

   [name@server ~]$ module load r/4.1
   

3. Start the R shell and install any required package. For example doParallel.

   [name@server ~]$ R --no-save
   

   > install.packages('doParallel', repos='http://cran.us.r-project.org')
   

4. Quit R (with q()) and close the remote session
5. The newly installed R packages should already be usable in a Notebook executed by the R kernel.

HyperText Transfer Protocol