JupyterNotebook

Introduction

Project Jupyter is an open source project, born out of the IPython Project, as it evolved to support interactive data science and scientific computing across all programming languages. The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text .

You can run Jupyter Notebook on the login node (not recommended) or the compute nodes (highly recommended). Note that login nodes impose various user- and process-based limits, so notebooks running there may be killed if they consume significant cpu-time or memory. To use a compute node you will have to submit a job requesting the number of CPUs (and, optionally, GPUs), the amount of memory, and the run time. Here, we give the instructions to submit a Jupyter Notebook job.

Install Jupyter Notebook

These instructions install Jupyter Notebook with the pip command in a Python virtual environment in your home directory. The following instructions are for Python 3.5.2, but you can also install the application for a different version by loading a different Python module.

1. Load the Python module:

   

   [name@server ~]$ module load python/3.5.2
   

2. Create a new Python virtual environment:

   

   [name@server ~]$ virtualenv $HOME/jupyter_py3
   

3. Activate your newly created Python virtual environment:

   

   [name@server ~]$ source $HOME/jupyter_py3/bin/activate
   

4. Install Jupyter into your virtual environment:

   

   (jupyter_py3)[name@server $] pip install jupyter
   

5. In your virtual environment, create a wrapper script that launches Jupyter notebook

   

   (jupyter_py3)[name@server $] echo -e '#!/bin/bash\nunset XDG_RUNTIME_DIR\njupyter notebook --ip $(hostname -f) --no-browser' > $VIRTUAL_ENV/bin/notebook.sh
   

6. Finally, make the script executable

   

   (jupyter_py3)[name@server $] chmod u+x $VIRTUAL_ENV/bin/notebook.sh
   

Install Extensions

Extensions allow you to add functionalities and modify the appearance of the Notebook application.

Jupyter Lmod

Jupyter Lmod is an extension that allows you to interact with environment modules before launching kernels. The extension use Lmod's Python interface to accomplish module-related tasks like loading, unloading, saving a collection, etc.

(jupyter_py3)[name@server $] pip install jupyterlmod
(jupyter_py3)[name@server $] jupyter nbextension install --py jupyterlmod --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension enable --py jupyterlmod --sys-prefix
(jupyter_py3)[name@server $] jupyter serverextension enable --py jupyterlmod --sys-prefix

RStudio Launcher

Jupyter can start an RStudio session that uses Jupyter's token authentication system. This extension adds an "RStudio Session" button to the New notebook menu.

(jupyter_py3)[name@server $] pip install nbserverproxy
(jupyter_py3)[name@server $] pip install git+https://github.com/cmd-ntrf/nbrsessionproxy
(jupyter_py3)[name@server $] jupyter serverextension enable --py nbserverproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension install --py nbrsessionproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension enable --py nbrsessionproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter serverextension enable --py nbrsessionproxy --sys-prefix

Activate the environment

Once you have installed Jupyter, each time you log in to the cluster you need only re-load the Python module associated with your environment:

[name@server ~]$ module load python/3.5.2

Then, activate the virtual environment in which you have installed Jupyter:

[name@server ~]$ source $HOME/jupyter_py3/bin/activate

RStudio Server (optional)

If you have installed the RStudio launcher extension and wish to use it, you will have to load the RStudio Server module.

(jupyter_py3)[name@server $] module load rstudio-server

Start the Notebook

To start the Notebook, submit an interactive job. Adjust the parameters based on your needs. See Running jobs for more information.

[name@server ~]$ salloc --time=1:0:0 --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1024M --account=def-yourpi srun notebook.sh
salloc: Granted job allocation 1422754
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.662 NotebookApp] 0 active kernels
[I 14:07:08.662 NotebookApp] The Jupyter Notebook is running at:
[I 14:07:08.663 NotebookApp] http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca
[I 14:07:08.663 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 14:07:08.669 NotebookApp]

Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca

Copy/paste the provided URL into your browser and enjoy your notebook.

Connect to the Notebook

To access the notebook running on a compute node from your web browser, you will need to create an SSH tunnel between the cluster and your computer since the compute nodes are not directly accessible from the Internet.

From Linux or MacOS X

On a Linux or MacOS X system we recommend using the Python package sshuttle.

On your computer, open a new terminal window, and run the following sshuttle command to create the tunnel

[name@my_computer $] sshuttle --dns -Nr userid@machine_name

From Windows

An SSH tunnel can be created from Windows using [MobaXTerm] as follows. Open two sessions in MobaXTerm.

1. The first session should be a connection to a cluster. Follow the instructions in Start the Notebook above to create a Jupyter notebook.
2. The second MobaXTerm session should be a local terminal.

Session 1: when you run the salloc, you will get something like this,

(jupyter_py3) [jnandez@gra-login1 ~]$ salloc --time=1:0:0 --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1024M --account=cc-debug srun $VIRTUAL_ENV/bin/notebook.sh
salloc: Pending job allocation 1060408
salloc: job 1060408 queued and waiting for resources
salloc: job 1060408 has been allocated resources
salloc: Granted job allocation 1060408
[I 12:53:31.208 NotebookApp] Loading lmod extension
[I 12:53:31.213 NotebookApp] Serving notebooks from local directory: /home/jnandez
[I 12:53:31.213 NotebookApp] 0 active kernels
[I 12:53:31.213 NotebookApp] The Jupyter Notebook is running at:
[I 12:53:31.213 NotebookApp] http://gra800.graham.sharcnet:8888/?token=9f31fae47604d65cf7706e706960a2f519b437d22f85eca1
[I 12:53:31.213 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 12:53:31.216 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://gra800.graham.sharcnet:8888/?token=9f31fae47604d65cf7706e706960a2f519b437d22f85eca1

Session 2: run the following command

[name@my_computer ]$  ssh -L 8888:gra800.graham.sharcnet:8888 jnandez@graham.computecanada.ca

The above command means that you will do a local port forwarding (-L), then it says that we will forward our local port 8888 to gra800.graham.sharcnet:8888. Note that you can pick another local port, we have used the same so we know that we are working remotely. Now open your browser and go to

http://localhost:8888/?token=9f31fae47604d65cf7706e706960a2f519b437d22f85eca1

The token is the same as printed by Session 1. You can also type http://localhost:8888, and there will be a prompt asking you for the token, which you will copy and paste from Session 1.

Shut down the Notebook

To shut down the Notebook server before the walltime limit, in the terminal that launched the interactive job, press Ctrl-C two times.