Jupyter Notebook

Introduction

"Project Jupyter is a non-profit, open-source project, born out of the IPython Project in 2014 as it evolved to support interactive data science and scientific computing across all programming languages."

"The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text."

You can run Jupyter Notebook on a compute node (highly recommended) or on a login node (not recommended). Note that login nodes impose various user- and process-based limits, so applications running there may be killed if they consume too much CPU time or memory. To use a compute node you will have to submit a job requesting the number of CPUs (and optionally GPUs), the amount of memory, and the run time. Here, we give instructions to submit a Jupyter Notebook job.

Certains de nos partenaires régionaux ont un portail JupyterHub pour éviter aux utilisateurs de créer leur propre configuration Jupyter Notebook. Pour plus d'information, consultez JupyterHub.

Installation

Ces directives permettent d'installer Jupyter Notebook avec la commande pip dans un un environnement virtuel | environnement virtuel Python dans votre répertoire home. Les directives sont pour la version 3.5.2 de Python, mais vous pouvez installer l'application pour d'autres versions en chargeant le module Python approprié.

1. Chargez le module Python.

   

   [name@server ~]$ module load python/3.5.2
   

2. Créez un nouvel environnement virtuel Python.

   

   [name@server ~]$ virtualenv $HOME/jupyter_py3
   

3. Activez votre nouvel environnement virtuel Python.

   

   [name@server ~]$ source $HOME/jupyter_py3/bin/activate
   

4. Installez Jupyter Notebook dans votre nouvel environnement virtuel Python.

   

   (jupyter_py3)[name@server $] pip install jupyter
   

5. Dans votre nouvel environnement virtuel, créez un script (wrapper) pour lancer Jupyter Notebook.

   

   (jupyter_py3)[name@server $] echo -e '#!/bin/bash\nunset XDG_RUNTIME_DIR\njupyter notebook --ip $(hostname -f) --no-browser' > $VIRTUAL_ENV/bin/notebook.sh
   

6. Enfin, rendez le script exécutable.

   

   (jupyter_py3)[name@server $] chmod u+x $VIRTUAL_ENV/bin/notebook.sh
   

Installer des extensions

Les extensions ajoutent des fonctionnalités et peuvent modifier l'interface utilisateur de l'application.

Jupyter Lmod

Jupyter Lmod est une extension permettant d'interagir avec les modules d'environnement avant le lancement des noyaux (kernels). L'extension utilise l'interface Python de Lmod pour accomplir des tâches reliées aux modules comme le chargement, le déchargement, la sauvegarde des collections, etc.

(jupyter_py3)[name@server $] pip install jupyterlmod
(jupyter_py3)[name@server $] jupyter nbextension install --py jupyterlmod --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension enable --py jupyterlmod --sys-prefix
(jupyter_py3)[name@server $] jupyter serverextension enable --py jupyterlmod --sys-prefix

RStudio Launcher

Jupyter Notebook can start an RStudio session that uses Jupyter Notebook's token authentication system. This extension adds an RStudio Session button to the Jupyter Notebook New menu.

(jupyter_py3)[name@server $] pip install nbserverproxy
(jupyter_py3)[name@server $] pip install git+https://github.com/cmd-ntrf/nbrsessionproxy
(jupyter_py3)[name@server $] jupyter serverextension enable --py nbserverproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension install --py nbrsessionproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension enable --py nbrsessionproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter serverextension enable --py nbrsessionproxy --sys-prefix

Activer l'environnement

Une fois que Jupyter Notebook est installé, vous n'aurez qu'à recharger le module Python associé à votre environnement lorsque vous vous connectez à la grappe.

[name@server ~]$ module load python/3.5.2

Activez ensuite l'environnement virtuel dans lequel Jupyter Notebook est installé.

[name@server ~]$ source $HOME/jupyter_py3/bin/activate

RStudio Server (optionnel)

If you have installed the RStudio launcher extension and wish to use it, you will have to load the RStudio Server module.

(jupyter_py3)[name@server $] module load rstudio-server

Lancer Jupyter Notebook

Pour lancer l'application, soumettez une tâche interactive. Ajustez les paramètres selon vos besoins. Pour plus d'information, consultez Exécuter des tâches.

[name@server ~]$ salloc --time=1:0:0 --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1024M --account=def-yourpi srun notebook.sh
salloc: Granted job allocation 1422754
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.662 NotebookApp] 0 active kernels
[I 14:07:08.662 NotebookApp] The Jupyter Notebook is running at:
[I 14:07:08.663 NotebookApp] http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca
[I 14:07:08.663 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 14:07:08.669 NotebookApp]

Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca

Se connecter à Jupyter Notebook

Puisque les nœuds de calcul ne sont pas directement accessibles par l'Internet, vous devez créer un tunnel SSH entre la grappe et votre poste de travail pour que votre fureteur web puisse avoir accès à Juniper Notebook opérant sur un nœud de calcul.

Sous Linux ou MacOS X

Nous recommandons le paquet Python sshuttle.

On your computer, open a new terminal window and run the following sshuttle command to create the tunnel.

[name@my_computer $] sshuttle --dns -Nr userid@machine_name

Then, copy and paste the provided URL into your browser. In the above example, this would be

 http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca

Sous Windows

An SSH tunnel can be created from Windows using MobaXTerm as follows.

Open two sessions in MobaXTerm.

• Session 1 should be a connection to a cluster. Follow the instructions in section Starting Jupyter Notebook.

• Session 2 should be a local terminal in which we will set up the SSH tunnel. Run the following command, substituting the node name from the URL you received in Session 1. Follow the instructions in section Starting Jupyter Notebook.

[name@my_computer ]$  ssh -L 8888:cdr544.int.cedar.computecanada.ca:8888 someuser@cedar.computecanada.ca

This command performs local port forwarding (-L). It forwards local port 8888 to cdr544.int.cedar.computecanada.ca:8888, which is the host name given when Jupyter Notebook was started.

Dans votre fureteur, entrez

 http://localhost:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca

Replace the token in this example with the one given to you in Session 1. You can also type http://localhost:8888 and there will be a prompt asking you for the token, which you can then copy and paste.

Fermer Jupyter Notebook

You can shut down the Jupyter Notebook server before the walltime limit by pressing Ctrl-C twice in the terminal that launched the interactive job.

If you used MobaXterm to create a tunnel, press Ctrl-D in Session 2 to shut down the tunnel.

Adding kernels

It is possible to add kernels for other programmming languages or Python versions different than the one running the Jupyter Notebook. Refer to Making kernels for Jupyter to learn more.

The installation of a new kernel is done in two steps.

1. Installation of the packages that will allow the language interpreter to communicate with Jupyter Notebook.
2. Creation of a file that will indicate to Jupyter Notebook how to initiate a communication channel with the language interpreter. This file is called a kernel spec file.

Each kernel spec file has to be created in its own subfolder inside a folder in your home directory with the following path ~/.local/share/jupyter/kernels. Jupyter Notebook does not create this folder, so the first step in all cases is to create it. You can use the following command.

[name@server ~]$ mkdir -p  ~/.local/share/jupyter/kernels

In the following sections, we provide a few examples of the kernel installation procedure.

Anaconda

Before installing an Anaconda kernel, make sure you have read the documentation and installed Anaconda.

1. Load the Anaconda module.
   

   [name@server ~]$ module load miniconda3
   

2. Optional: Activate a specific conda virtual environment.
   

   [name@server ~]$ source activate <your env>
   

3. Install the ipykernel library.
   

   [name@server ~]$ conda install ipykernel
   

4. Generate the kernel spec file.
   

   [name@server ~]$ python -m ipykernel install --user --name <unique identifier without white space> --display-name "My Anaconda 3 Kernel"
   

5. Optional: Deactivate the virtual environment.
   

   [name@server ~]$ source deactivate
   

For more information, see the ipykernel documentation.

Julia

1. Load the Julia module.
   

   [name@server ~]$ module load julia
   

2. Activate the Jupyter Notebook virtual environment.
   

   [name@server ~]$ source $HOME/jupyter_py3/bin/activate
   

3. Install IJulia.
   

   [name@server ~]$ echo 'Pkg.add("IJulia")' | julia
   

For more information, see the IJulia documentation.

R

1. Load the R module.
   

   [name@server ~]$ module load r
   

2. Activate the Jupyter Notebook virtual environment.
   

   [name@server ~]$ source $HOME/jupyter_py3/bin/activate
   

3. Install the R kernel dependencies.
   

   [name@server ~]$ R -e "install.packages(c('crayon', 'pbdZMQ', 'devtools'), repos='http://cran.us.r-project.org')"
   

4. Install the R kernel.
   

   [name@server ~]$ R -e "devtools::install_github(paste0('IRkernel/', c('repr', 'IRdisplay', 'IRkernel')))"
   

5. Install the R kernel spec file.
   

   [name@server ~]$ R -e "IRkernel::installspec()"
   

For more information, see the IRKernel documentation.