JupyterNotebook

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

Introduction

Jupyter notebook comes in one Python model on Graham. You can get it working on the login node (not recommended), and the compute nodes (highly recommended). Note that Login nodes impose various user- and process-based limits, so notebooks running there may be killed if they consume significant cpu-time or memory. You will have to submit a job requesting the # of CPU (or even GPU), amount of memory and runtime. Here, we give the instructions to submit a Jupyter job.

Installing Jupyter Notebook

These instructions install Jupyter into your home directory. To install Jupyter we will use the pip command and install it into a Python virtual environments. The below instructions install for Python 3.5.2. The below instructions install for Python 3.5.2 but you can also install for Python 3.5.Y or 2.7.X by loading a different Python module.

Load the Python module:

[name@server ~]$ module load python/3.5.2

Create a new python virtual environment:

[name@server ~]$ virtualenv $HOME/jupyter_py3

Activate your newly created python virtual environment:

[name@server ~]$ source $HOME/jupyter_py3/bin/activate

Install Jupyter into your newly created virtual environment:

(jupyter_py3)[name@server $] pip install jupyter

In your virtual environment, create a wrapper script that launches Jupyter notebook

(jupyter_py3)[name@server $] echo -e '#!/bin/bash\nunset XDG_RUNTIME_DIR\njupyter notebook --ip $(hostname) --no-browser' > $VIRTUAL_ENV/bin/notebook.sh

Finally, make the script executable

(jupyter_py3)[name@server $] chmod u+x $VIRTUAL_ENV/bin/notebook.sh

Install Jupyter Lmod

[Lmod] is an extension that allows user to interact with environment modules before launching kernels. The extension use Lmod's Python interface to accomplish module related task like loading, unloading, saving collection, etc.

(jupyter_py3)[name@server $] pip install jupyterlmod
(jupyter_py3)[name@server $] jupyter nbextension install --py jupyterlmod --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension enable --py jupyterlmod --sys-prefix
(jupyter_py3)[name@server $] jupyter serverextension enable --py jupyterlmod --sys-prefix

Connecting to a manually spawned Jupyter Notebook

Create a Tunnel

To access the notebook running on a compute node from your web browser, you will need to create a tunnel between the cluster and your computer since the compute nodes are not directly accessible from the Internet. To create that tunnel, we recommend the usage of the Python package sshuttle.

On your computer, open a new terminal window, and run the following sshuttle command to create the tunnel

[name@my_computer $] sshuttle --dns -Nr userid@machine_name

Activate the Environment

On the cluster, activate the virtual environment in which you have installed Jupyter.

[name@server ~]$ source $HOME/jupyter_py3/bin/activate

Start the Notebook

To start the Notebook, submit an interactive job. Adjust the parameters based on your needs. See Running jobs for more information.

[name@server ~]$ salloc --time=1:0:0 --ntasks=1 --ncpus-per-task=2 --mem-per-cpu=1024M --account=def-yourpi srun notebook.sh
salloc: Granted job allocation 1422754
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.662 NotebookApp] 0 active kernels
[I 14:07:08.662 NotebookApp] The Jupyter Notebook is running at:
[I 14:07:08.663 NotebookApp] http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca
[I 14:07:08.663 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 14:07:08.669 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca

Copy/paste the provided URL into your browser and enjoy your notebook.

Shutdown the Notebook

To shutdown the Notebook server before the walltime limit, in the terminal that launched the interactive job, press Ctrl-C two times.