JupyterNotebook

Introduction

Project Jupyter is an open source project, born out of the IPython Project, as it evolved to support interactive data science and scientific computing across all programming languages.

Jupyter notebook comes in one Python model on Graham. You can get it working on the login node (not recommended), and the compute nodes (highly recommended). Note that login nodes impose various user- and process-based limits, so notebooks running there may be killed if they consume significant cpu-time or memory. To use a compute node you will have to submit a job requesting the number of CPUs (and, optionally, GPUs), the amount of memory, and the run time. Here, we give the instructions to submit a Jupyter job.

Installing Jupyter Notebook

These instructions install Jupyter into your home directory. To install Jupyter we will use the pip command and install it into a Python virtual environment. The below instructions install for Python 3.5.2 but you can also install for Python 3.5.Y or 2.7.X by loading a different Python module.

Load the Python module:

[name@server ~]$ module load python/3.5.2

Create a new Python virtual environment:

[name@server ~]$ virtualenv $HOME/jupyter_py3

Activate your newly created Python virtual environment:

[name@server ~]$ source $HOME/jupyter_py3/bin/activate

Install Jupyter into your virtual environment:

(jupyter_py3)[name@server $] pip install jupyter

In your virtual environment, create a wrapper script that launches Jupyter notebook

(jupyter_py3)[name@server $] echo -e '#!/bin/bash\nunset XDG_RUNTIME_DIR\njupyter notebook --ip $(hostname -f) --no-browser' > $VIRTUAL_ENV/bin/notebook.sh

Finally, make the script executable

(jupyter_py3)[name@server $] chmod u+x $VIRTUAL_ENV/bin/notebook.sh

Installing Extensions

Extensions allow you to add functionalities and modify the appearance of the Notebook application.

Jupyter Lmod

Jupyter Lmod is an extension that allows you to interact with environment modules before launching kernels. The extension use Lmod's Python interface to accomplish module-related tasks like loading, unloading, saving a collection, etc.

(jupyter_py3)[name@server $] pip install jupyterlmod
(jupyter_py3)[name@server $] jupyter nbextension install --py jupyterlmod --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension enable --py jupyterlmod --sys-prefix
(jupyter_py3)[name@server $] jupyter serverextension enable --py jupyterlmod --sys-prefix

RStudio Launcher

Jupyter can start an RStudio session that uses Jupyter's token authentication system. This extension adds an "RStudio Session" button to the New notebook menu.

(jupyter_py3)[name@server $] pip install nbserverproxy
(jupyter_py3)[name@server $] pip install git+https://github.com/cmd-ntrf/nbrsessionproxy
(jupyter_py3)[name@server $] jupyter serverextension enable --py nbserverproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension install --py nbrsessionproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension enable --py nbrsessionproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter serverextension enable --py nbrsessionproxy --sys-prefix

Connecting to a manually spawned Jupyter Notebook

Create a tunnel in Linux or MacOS X

To access the notebook running on a compute node from your web browser, you will need to create a tunnel between the cluster and your computer since the compute nodes are not directly accessible from the Internet. To create that tunnel, we recommend the usage of the Python package sshuttle.

On your computer, open a new terminal window, and run the following sshuttle command to create the tunnel

[name@my_computer $] sshuttle --dns -Nr userid@machine_name

Activate the environment

On the cluster, load the Python module associated with your environment:

[name@server ~]$ module load python/3.5.2

Then, activate the virtual environment in which you have installed Jupyter:

[name@server ~]$ source $HOME/jupyter_py3/bin/activate

RStudio Server (optional)

If you have installed the RStudio launcher extension and wish to use it, you will have to load the RStudio Server module.

[name@server ~]$ module load rstudio-server

Start the Notebook

To start the Notebook, submit an interactive job. Adjust the parameters based on your needs. See Running jobs for more information.

[name@server ~]$ salloc --time=1:0:0 --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1024M --account=def-yourpi srun notebook.sh
salloc: Granted job allocation 1422754
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.662 NotebookApp] 0 active kernels
[I 14:07:08.662 NotebookApp] The Jupyter Notebook is running at:
[I 14:07:08.663 NotebookApp] http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca
[I 14:07:08.663 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 14:07:08.669 NotebookApp]

Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca

Copy/paste the provided URL into your browser and enjoy your notebook.

Shut down the Notebook

To shut down the Notebook server before the walltime limit, in the terminal that launched the interactive job, press Ctrl-C two times.