JupyterNotebook

Introduction

Project Jupyter is an open source project, born out of the IPython Project, as it evolved to support interactive data science and scientific computing across all programming languages.

Jupyter notebook comes in one Python model on Graham. You can get it working on the login node (not recommended), and the compute nodes (highly recommended). Note that login nodes impose various user- and process-based limits, so notebooks running there may be killed if they consume significant cpu-time or memory. To use a compute node you will have to submit a job requesting the number of CPUs (and, optionally, GPUs), the amount of memory, and the run time. Here, we give the instructions to submit a Jupyter job.

Installing Jupyter Notebook

These instructions install Jupyter into your home directory. To install Jupyter we will use the pip command and install it into a Python virtual environment. The below instructions install for Python 3.5.2 but you can also install for Python 3.5.Y or 2.7.X by loading a different Python module.

Load the Python module:

[name@server ~]$ module load python/3.5.2

Create a new Python virtual environment:

[name@server ~]$ virtualenv $HOME/jupyter_py3

Activate your newly created Python virtual environment:

[name@server ~]$ source $HOME/jupyter_py3/bin/activate

Install Jupyter into your virtual environment:

(jupyter_py3)[name@server $] pip install jupyter

In your virtual environment, create a wrapper script that launches Jupyter notebook

(jupyter_py3)[name@server $] echo -e '#!/bin/bash\nunset XDG_RUNTIME_DIR\njupyter notebook --ip $(hostname -f) --no-browser' > $VIRTUAL_ENV/bin/notebook.sh

Finally, make the script executable

(jupyter_py3)[name@server $] chmod u+x $VIRTUAL_ENV/bin/notebook.sh

Installing Extensions

Extensions allow you to add functionalities and modify the appearance of the Notebook application.

Jupyter Lmod

Jupyter Lmod is an extension that allows you to interact with environment modules before launching kernels. The extension use Lmod's Python interface to accomplish module-related tasks like loading, unloading, saving a collection, etc.

(jupyter_py3)[name@server $] pip install jupyterlmod
(jupyter_py3)[name@server $] jupyter nbextension install --py jupyterlmod --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension enable --py jupyterlmod --sys-prefix
(jupyter_py3)[name@server $] jupyter serverextension enable --py jupyterlmod --sys-prefix

RStudio Launcher

Jupyter can start an RStudio session that uses Jupyter's token authentication system. This extension adds an "RStudio Session" button to the New notebook menu.

(jupyter_py3)[name@server $] pip install nbserverproxy
(jupyter_py3)[name@server $] pip install git+https://github.com/cmd-ntrf/nbrsessionproxy
(jupyter_py3)[name@server $] jupyter serverextension enable --py nbserverproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension install --py nbrsessionproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension enable --py nbrsessionproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter serverextension enable --py nbrsessionproxy --sys-prefix

Connecting to a manually spawned Jupyter Notebook

Create a tunnel

To access the notebook running on a compute node from your web browser, you will need to create a tunnel between the cluster and your computer since the compute nodes are not directly accessible from the Internet. To create that tunnel (on a Linux or MaxOS X system), we recommend the usage of the Python package sshuttle.

On your computer, open a new terminal window, and run the following sshuttle command to create the tunnel

[name@my_computer $] sshuttle --dns -Nr userid@machine_name

Windows

We have tested MobaXTerm on Windows 10 using ssh as follows. Open 2 sessions, (1) will have the above instructions, and (2) will have the ssh port-forwarding.

Session 1: when you run the salloc, you will get something like this,

(jupyter_py3) [jnandez@gra-login1 ~]$ salloc --time=1:0:0 --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1024M --account=cc-debug srun $VIRTUAL_ENV/bin/notebook.sh
salloc: Pending job allocation 1060408
salloc: job 1060408 queued and waiting for resources
salloc: job 1060408 has been allocated resources
salloc: Granted job allocation 1060408
[I 12:53:31.208 NotebookApp] Loading lmod extension
[I 12:53:31.213 NotebookApp] Serving notebooks from local directory: /home/jnandez
[I 12:53:31.213 NotebookApp] 0 active kernels
[I 12:53:31.213 NotebookApp] The Jupyter Notebook is running at:
[I 12:53:31.213 NotebookApp] http://gra800.graham.sharcnet:8888/?token=9f31fae47604d65cf7706e706960a2f519b437d22f85eca1
[I 12:53:31.213 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 12:53:31.216 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://gra800.graham.sharcnet:8888/?token=9f31fae47604d65cf7706e706960a2f519b437d22f85eca1

Session 2: run the following command

[name@my_computer ]$  ssh -L 8888:gra800.graham.sharcnet:8888 jnandez@graham.computecanada.ca

The above command means that you will do a local port forwarding (-L), then it says that we will forward our local port 8888 to gra800.graham.sharcnet:8888. Note that you can pick another local port, we have used the same so we know that we are working remotely. Now open your browser and go to

http://localhost:8888/?token=9f31fae47604d65cf7706e706960a2f519b437d22f85eca1

The token is the same as printed by Session 1. You can also type http://localhost:8888, and there will be a prompt asking you for the token, which you will copy and paste from Session 1.

Activate the environment

On the cluster, load the Python module associated with your environment:

[name@server ~]$ module load python/3.5.2

Then, activate the virtual environment in which you have installed Jupyter:

[name@server ~]$ source $HOME/jupyter_py3/bin/activate

RStudio Server (optional)

If you have installed the RStudio launcher extension and wish to use it, you will have to load the RStudio Server module.

[name@server ~]$ module load rstudio-server

Start the Notebook

To start the Notebook, submit an interactive job. Adjust the parameters based on your needs. See Running jobs for more information.

[name@server ~]$ salloc --time=1:0:0 --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1024M --account=def-yourpi srun notebook.sh
salloc: Granted job allocation 1422754
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.662 NotebookApp] 0 active kernels
[I 14:07:08.662 NotebookApp] The Jupyter Notebook is running at:
[I 14:07:08.663 NotebookApp] http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca
[I 14:07:08.663 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 14:07:08.669 NotebookApp]

Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca

Copy/paste the provided URL into your browser and enjoy your notebook.

Shut down the Notebook

To shut down the Notebook server before the walltime limit, in the terminal that launched the interactive job, press Ctrl-C two times.