JupyterHub

JupyterHub is the best way to serve Jupyter Notebook for multiple users. It can be used in a class of students, a corporate data science group or scientific research group.

JupyterHub provides a preconfigured version of JupyterLab and/or Jupyter Notebook; for more configuration options, please check the Jupyter page.

Running notebooks

Jupyter Lab and notebooks are meant for short interactive tasks such as testing, debugging or quickly visualize data (few minutes). Running longer analysis must be done in an non-interactive job (sbatch).

Alliance initiatives

Some regional initiatives offer access to computing resources through JupyterHub.

JupyterHub on clusters

On the following clusters^‡, use your Alliance username and password to connect to JupyterHub:

JupyterHub	Comments
Béluga	Provides access to JupyterLab servers spawned through jobs on the Béluga cluster.
Cedar	Provides access to JupyterLab servers spawned through jobs on the Cedar cluster.
Narval	Provides access to JupyterLab servers spawned through jobs on the Narval cluster.
Niagara	This is a node which has been designated as a Jupyter Hub and it can run Jupyter Notebook sessions. To learn more, see the SciNet JupyterHub wiki page.
Graham	Provides access to JupyterLab servers spawned through jobs on the Graham cluster.

^‡ Note that the compute nodes running the Jupyter kernels do not have internet access. This means that you can only transfer files from/to your own computer; you cannot download code or data from the internet (e.g. cannot do "git clone", cannot do "pip install" if the wheel is absent from our wheelhouse). You may also have problems if your code performs downloads or uploads (e.g. in machine learning where downloading data from the code is often done).

JupyterHub for universities and schools

• The Pacific Institute for the Mathematical Sciences in collaboration with the Alliance and Cybera offer cloud-based hubs to universities and schools. Each institution can have its own hub where users authenticate with their credentials from that institution. The hubs are hosted on Alliance clouds and are essentially for training purposes. Institutions interested in obtaining their own hub can visit Syzygy.

Server options

Server Options form on Béluga's JupyterHub

Once logged in, depending on the configuration of JupyterHub, the user's web browser is redirected to either a) a previously launched Jupyter server, b) a new Jupyter server with default options, or c) a form that allows a user to set different options for their Jupyter server before pressing the Start button. In all cases, it is equivalent to accessing requested resources via an interactive job on the corresponding cluster.

Important: On each cluster, only one interactive job at a time gets a priority increase in order to start in a few seconds or minutes. That includes salloc, srun and JupyterHub jobs. If you already have another interactive job running on the cluster hosting JupyterHub, your new Jupyter session may never start before the time limit of 5 minutes.

Compute resources

For example, Server Options available on Béluga's JupyterHub are:

• Account to be used: any def-*, rrg-*, rpp-* or ctb-* account a user has access to
• Time (hours) required for the session
• Number of (CPU) cores that will be reserved on a single node
• Memory (MB) limit for the entire session
• (Optional) GPU configuration: at least one GPU
• User interface (see below)

User interface

While JupyterHub allows each user to use one Jupyter server at a time on each hub, there can be multiple options under User interface:

• Jupyter Notebook (classic interface): Even though it offers many functionalities, the community is moving towards JupyterLab, which is a better platform that offers many more features.
• JupyterLab (modern interface): This is the most recommended Jupyter user interface for interactive prototyping and data visualization.
• Terminal (for a single terminal only): It gives access to a terminal connected to a remote account, which is comparable to connecting to a server through an SSH connection.

Note: JupyterHub could also have been configured to force a specific user interface. This is usually done for special events.

JupyterLab

JupyterLab is now the recommended general-purpose user interface to use on a JupyterHub. From a JupyterLab server, you can manage your remote files and folders, and you can launch Jupyter applications like a terminal, (Python 3) notebooks, RStudio and a Linux desktop.

The JupyterLab interface

When JupyterLab is ready to be used, the interface has multiple panels.

Default home tab when JupyterLab is loaded

Menu bar on top

• In the File menu:
  • Hub Control Panel: if you want to manually stop the JupyterLab server and the corresponding job on the cluster. This is useful when you want to start a new JupyterLab server with more or less resources.
  • Log Out: the JupyterHub session will end, which will also stop the JupyterLab server and the corresponding job on the cluster.
• Most other menu items are related to notebooks and Jupyter applications.

Tool selector on left

• File Browser (folder icon):
  • This is where you can browse in your home, project and scratch spaces.
  • It is also possible to upload files.
• Running Terminals and Kernels (stop icon):
  • To stop kernel sessions and terminal sessions
• Commands
• Property Inspector
• Open Tabs:
  • To navigate between application tabs.
  • To close application tabs (the corresponding kernels remain active).

Loaded modules and available modules

• Software (blue diamond sign):
  • Alliance modules can be loaded and unloaded in the JupyterLab session. Depending on the modules loaded, icons directing to the corresponding Jupyter applications will appear in the Launcher tab.
  • The search box can search for any available module and show the result in the Available Modules subpanel. Note: Some modules are hidden until their dependency is loaded: we recommend that you first look for a specific module with module spider module_name from a terminal.
  • The next subpanel is the list of Loaded Modules in the whole JupyterLab session. Note: While python and ipython-kernel modules are loaded by default, additional modules must be loaded before launching some other applications or notebooks. For example: scipy-stack.
  • The last subpanel is the list of Available modules, similar to the output of module avail. By clicking on a module's name, detailed information about the module is displayed. By clicking on the Load link, the module will be loaded and added to the Loaded Modules list.

Applications area on right

• The Launcher tab is open by default.
  • It contains all available Jupyter applications and notebooks, depending on which modules are loaded

Status bar at the bottom

• By clicking on the icons, this brings you to the Running Terminals and Kernels tool.

Prebuilt applications

JupyterLab offers access to a terminal, an IDE (Desktop), a Python console and different options to create text and markdown files. This section presents only the main supported Jupyter applications that work with our software stack.

Command line interpreters

Julia console launcher button

Python console launcher button

Terminal launcher button

Julia console

To enable the Julia 1.x console launcher, an ijulia-kernel module needs to be loaded. When launched, a Julia interpreter is presented in a new JupyterLab tab.

Python console

The Python 3.x console launcher is available by default in a new JupyterLab session. When launched, a Python 3 interpreter is presented in a new JupyterLab tab.

Terminal

This application launcher will open a terminal in a new JupyterLab tab:

• The terminal runs a (Bash) shell on the remote compute node without the need of an SSH connection.
  • Gives access to the remote filesystems (/home, /project, /scratch).
  • Allows running compute tasks.
• The terminal allows copy-and-paste operations of text:
  • Copy operation: select the text, then press Ctrl+C.
    • Note: Usually, Ctrl+C is used to send a SIGINT signal to a running process, or to cancel the current command. To get this behaviour in JupyterLab's terminal, click on the terminal to deselect any text before pressing Ctrl+C.
  • Paste operation: press Ctrl+V.

Available notebook kernels

Julia notebook

To enable the Julia 1.x notebook launcher, an ijulia-kernel module needs to be loaded. When launched, a Julia notebook is presented in a new JupyterLab tab.

Python notebook

Searching for scipy-stack modules

If any of the following scientific Python packages is required by your notebook, before you open this notebook, you must load the scipy-stack module from the JupyterLab Softwares tool:

• ipython, ipython_genutils, ipykernel, ipyparallel
• matplotlib
• numpy
• pandas
• scipy
• Other notable packages are Cycler, futures, jupyter_client, jupyter_core, mpmath, pathlib2, pexpect, pickleshare, ptyprocess, pyzmq, simplegeneric, sympy, tornado, traitlets.
• And many more (click on the scipy-stack module to see all Included extensions).

Note: You may also install needed packages by running for example the following command inside a cell: !pip install --no-index numpy.

• For some packages (like plotly, for example), you may need to restart the notebook's kernel before importing the package.
• The installation of packages in the default Python kernel environment is temporary to the lifetime of the JupyterLab session; you will have to reinstall these packages the next time you start a new JupyterLab session. For a persistent Python environment, you must configure a custom Python kernel.

To open an existing Python notebook:

• Go back to the File Browser.
• Browse to the location of the *.ipynb file.
• Double-click on the *.ipynb file.
  • This will open the Python notebook in a new JupyterLab tab.
  • An IPython kernel will start running in the background for this notebook.

To open a new Python notebook in the current File Browser directory:

• Click on the Python 3.x launcher under the Notebook section.
  • This will open a new Python 3 notebook in a new JupyterLab tab.
  • A new IPython kernel will start running in the background for this notebook.

Other applications

OpenRefine

OpenRefine launcher button

To enable the OpenRefine application launcher, an openrefine module needs to be loaded. Depending on the software environment version, the latest version of OpenRefine should be loaded:

• with StdEnv/2020, load module: openrefine/3.4.1
• with StdEnv/2018.3, load module: openrefine/3.3

This OpenRefine launcher will open or reopen an OpenRefine interface in a new web browser tab.

• It is possible to reopen an active OpenRefine session after the web browser tab was closed.
• The OpenRefine session will end when the JupyterLab session ends.

RStudio

RStudio launcher button

To enable the RStudio application launcher in StdEnv/2020, load the module: rstudio-server

With older software environment, you should load the following two modules (r is loaded automatically):

• with StdEnv/2018.3, load modules: gcc/7.3.0, rstudio-server/1.2.1335
• with StdEnv/2016.4, load modules: gcc/7.3.0, rstudio-server/1.2.1335

This RStudio launcher will open or reopen an RStudio interface in a new web browser tab.

• It is possible to reopen an active RStudio session after the web browser tab was closed.
• The RStudio session will end when the JupyterLab session ends.

VS Code

VS Code launcher button

To enable the VS Code (Visual Studio Code) application launcher, a code-server module needs to be loaded. Depending on the software environment version, the latest version of VS Code should be loaded:

• with StdEnv/2020, load module: code-server/3.12.0
• with StdEnv/2018.3, load module: code-server/3.4.1

This VS Code launcher will open or reopen the VS Code interface in a new web browser tab.

• For a new session, the VS Code session can take up to 3 minutes to complete its startup.
• It is possible to reopen an active VS Code session after the web browser tab was closed.
• The VS Code session will end when the JupyterLab session ends.

Desktop

Desktop launcher button

This Desktop launcher will open or reopen a remote Linux desktop interface in a new web browser tab:

• This is equivalent to running a VNC server on a compute node, then creating an SSH tunnel and finally using a VNC client, but you need nothing of all this with JupyterLab!
• It is possible to reopen an active desktop session after the web browser tab was closed.
• The desktop session will end when the JupyterLab session ends.

Running notebooks as Python scripts

1. From the console, or in a new notebook cell, install nbconvert :

!pip install --no-index nbconvert

2. Convert your notebooks to Python scripts

!jupyter nbconvert --to python my-current-notebook.ipynb

3. Create your non-interactive submission script, and submit it.

In your submission script, run your converted notebook with:

python my-current-notebook.py

And submit your non-interactive job:

[name@server ~]$ sbatch my-submit.sh

Possible error messages

Most JupyterHub errors are caused by the underlying job scheduler which is either unresponsive or not able to find appropriate resources for your session. For example:

• A "time-out" error message when starting a JupyterLab session:
  • Just like any interactive job on any cluster, a longer requested time can cause a longer wait time in queue.
  • There may be no available interactive node at the moment; try again later.