JupyterNotebook: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
mNo edit summary
 
(136 intermediate revisions by 15 users not shown)
Line 1: Line 1:
<languages />
<languages />
{{Warning|title=Advanced material|content=This page is for advanced users. Please see [[JupyterHub]] instead.}}


<translate>
<translate>
Line 5: Line 7:


<!--T:38-->
<!--T:38-->
[http://jupyter.org/about.html Project Jupyter] is an open source project, born out of the [https://ipython.org/ IPython Project], as it evolved to support interactive data science and scientific computing across all programming languages. The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text <ref>http://www.jupyter.org/, The Jupyter Notebook</ref>.
"Project Jupyter is a non-profit, open-source project, born out of the IPython Project in 2014 as it evolved to support interactive data science and scientific computing across all programming languages."<ref>http://jupyter.org/about.html</ref>
 
<!--T:51-->
"The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text."<ref>http://www.jupyter.org/</ref>


<!--T:2-->
<!--T:2-->
You can run Jupyter Notebook on the login node (not recommended) or the compute nodes (highly recommended). Note that login nodes impose various user- and process-based limits, so notebooks running there may be killed if they consume significant cpu-time or memory. To use a compute node you will have to submit a job requesting the number of CPUs (and, optionally, GPUs), the amount of memory, and the run time. Here, we give the instructions to submit a Jupyter Notebook job.
You can run Jupyter Notebook on a compute node or on a login node (not recommended). Note that login nodes impose various user- and process-based limits, so applications running there may be killed if they consume too much CPU time or memory. To use a compute node you will have to submit a job requesting the number of CPUs (and optionally GPUs), the amount of memory, and the run time. Here, we give instructions to submit a Jupyter Notebook job.
 
<!--T:50-->
'''Other information:'''
* Since Jupyter Notebook is the older Jupyter interface, please consider installing '''[[Advanced_Jupyter_configuration|JupyterLab]]''' instead.
* If you are instead looking for a preconfigured Jupyter environment, please see the '''[[Jupyter]]''' page.


== Install Jupyter Notebook == <!--T:3-->
== Installing Jupyter Notebook == <!--T:3-->


<!--T:4-->
<!--T:4-->
These instructions install Jupyter Notebook with the <code>pip</code> command in a [[Python#Creating_and_using_a_virtual_environment | Python virtual environment]] in your home directory. The following instructions are for Python 3.5.2, but you can also install the application for a different version by loading a different Python module.
These instructions install Jupyter Notebook with the <code>pip</code> command in a [[Python#Creating_and_using_a_virtual_environment | Python virtual environment]] in your home directory. The following instructions are for Python 3.6, but you can also install the application for a different version by loading a different Python module.


<!--T:5-->
<!--T:5-->
#Load the Python module:
<ol>
#:{{Command|module load python/3.5.2}}
<li>Load the Python module.
#Create a new Python virtual environment:
{{Command2|module load python/3.7}}
#:{{Command|virtualenv $HOME/jupyter_py3}}
<li>Create a new Python virtual environment.
#Activate your newly created Python virtual environment:
{{Command2|virtualenv $HOME/jupyter_py3}}
#:{{Command|source $HOME/jupyter_py3/bin/activate}}
<li>Activate your newly created Python virtual environment.
#Install Jupyter into your virtual environment:
{{Command2|source $HOME/jupyter_py3/bin/activate}}
#:{{Command
<li>Install Jupyter Notebook in your new virtual environment.
|prompt=(jupyter_py3)[name@server $]
{{Commands2
|pip install jupyter}}
|prompt=(jupyter_py3) [name@server ~]$
#In your virtual environment, create a wrapper script that launches Jupyter notebook
|pip install --no-index --upgrade pip
#:{{Command
|pip install --no-index jupyter
|prompt=(jupyter_py3)[name@server $]
}}
|echo -e '#!/bin/bash\nunset XDG_RUNTIME_DIR\njupyter notebook --ip $(hostname -f) --no-browser' > $VIRTUAL_ENV/bin/notebook.sh
<li>In the virtual environment, create a wrapper script that launches Jupyter Notebook.
{{Command2
|prompt=(jupyter_py3)_[name@server ~]$
|echo -e '#!/bin/bash\nexport JUPYTER_RUNTIME_DIR{{=}}$SLURM_TMPDIR/jupyter\njupyter notebook --ip $(hostname -f) --no-browser' > $VIRTUAL_ENV/bin/notebook.sh
}}
}}
#Finally, make the script executable
<li>Finally, make the script executable.
#:{{Command
{{Command2
|prompt=(jupyter_py3)[name@server $]
|prompt=(jupyter_py3)_[name@server ~]$
|chmod u+x $VIRTUAL_ENV/bin/notebook.sh
|chmod u+x $VIRTUAL_ENV/bin/notebook.sh
}}
}}
</ol>


== Install Extensions == <!--T:10-->
== Installing extensions == <!--T:10-->


<!--T:11-->
<!--T:11-->
Extensions allow you to add functionalities and modify the appearance of the Notebook application.  
Extensions allow you to add functionalities and modify the application’s user interface.  


=== Jupyter Lmod === <!--T:12-->
=== Jupyter Lmod === <!--T:12-->


<!--T:13-->
<!--T:13-->
[https://github.com/cmd-ntrf/jupyter-lmod| Jupyter Lmod] is an extension that allows you to interact with environment modules before launching kernels. The extension use Lmod's Python interface to accomplish module-related tasks like loading, unloading, saving a collection, etc.
[https://github.com/cmd-ntrf/jupyter-lmod Jupyter Lmod] is an extension that allows you to interact with environment modules before launching kernels. The extension uses the Lmod's Python interface to accomplish module-related tasks like loading, unloading, saving a collection, etc.


<!--T:14-->
{{Commands2
{{Commands
|prompt=(jupyter_py3)_[name@server ~]$
|prompt=(jupyter_py3)[name@server $]
|pip install jupyterlmod
|pip install jupyterlmod
|jupyter nbextension install --py jupyterlmod --sys-prefix
|jupyter nbextension install --py jupyterlmod --sys-prefix
Line 55: Line 68:
|jupyter serverextension enable --py jupyterlmod --sys-prefix
|jupyter serverextension enable --py jupyterlmod --sys-prefix
}}
}}
=== Proxy web services === <!--T:14-->
<!--T:75-->
[https://github.com/jupyterhub/nbserverproxy nbserverproxy] enables users to reach arbitrary web services running within their spawned Jupyter server. This is useful to access web services that are listening only on a port of the localhost like [https://www.tensorflow.org/programmers_guide/summaries_and_tensorboard TensorBoard].
{{Commands2
|prompt=(jupyter_py3)_[name@server ~]$
|pip install nbserverproxy
|jupyter serverextension enable --py nbserverproxy --sys-prefix
}}
==== Example ==== <!--T:71-->
<!--T:72-->
In Jupyter, a user starts a web service via 'Terminal' in the ''New'' dropdown list:
<!--T:73-->
{{Command2
|tensorboard --port{{=}}8008
}}
<!--T:74-->
The service is proxied off of /proxy/ at https://address.of.notebook.server/user/theuser/proxy/8008.


=== RStudio Launcher === <!--T:15-->
=== RStudio Launcher === <!--T:15-->


<!--T:16-->
<!--T:16-->
Jupyter can start an RStudio session that uses Jupyter's token authentication system. This extension adds an "RStudio Session" button to the New notebook menu.
Jupyter Notebook can start an RStudio session that uses Jupyter Notebook's token authentication system. RStudio Launcher adds an ''RStudio Session'' option to the Jupyter Notebook ''New'' dropdown list.
 
<!--T:81-->
'''Note:''' the installation procedure below only works with the <code>StdEnv/2016.4</code> and <code>StdEnv/2018.3</code> software environments.


<!--T:17-->
<!--T:17-->
{{Commands
{{Commands2
|prompt=(jupyter_py3)[name@server $]
|prompt=(jupyter_py3)_[name@server ~]$
|pip install nbserverproxy  
|pip install nbserverproxy  
|pip install git+https://github.com/cmd-ntrf/nbrsessionproxy
|pip install https://github.com/jupyterhub/nbrsessionproxy/archive/v0.8.0.zip
|jupyter serverextension enable --py nbserverproxy --sys-prefix
|jupyter serverextension enable --py nbserverproxy --sys-prefix
|jupyter nbextension install --py nbrsessionproxy --sys-prefix
|jupyter nbextension install --py nbrsessionproxy --sys-prefix
Line 72: Line 110:
}}
}}


== Activate the environment == <!--T:23-->
== Activating the environment == <!--T:23-->


<!--T:24-->
<!--T:24-->
Once you have installed Jupyter, each time you log in to the cluster you need only re-load the Python module associated with your environment:
Once you have installed Jupyter Notebook, you need only reload the Python module associated with your environment when you log into the cluster.


<!--T:25-->
<!--T:25-->
{{Command|module load python/3.5.2}}
{{Command2|module load python/3.7}}


<!--T:26-->
<!--T:26-->
Then, activate the virtual environment in which you have installed Jupyter:
Then, activate the virtual environment in which you have installed Jupyter Notebook.


<!--T:27-->
<!--T:27-->
{{Command|source $HOME/jupyter_py3/bin/activate}}
{{Command2|source $HOME/jupyter_py3/bin/activate}}


=== RStudio Server (optional) === <!--T:28-->  
=== RStudio Server (optional) === <!--T:28-->  


<!--T:29-->
<!--T:29-->
If you have installed the RStudio launcher extension and wish to use it, you will have to load the RStudio Server module.
To use [[#RStudio_Launcher | RStudio Launcher]], load the RStudio Server module.


<!--T:30-->
<!--T:30-->
{{Command|prompt=(jupyter_py3)[name@server $]
{{Command2|prompt=(jupyter_py3)_[name@server ~]$
|module load rstudio-server}}
|module load rstudio-server}}


== Start the Notebook == <!--T:31-->
== Starting Jupyter Notebook == <!--T:31-->


<!--T:32-->
<!--T:32-->
To start the Notebook, submit an interactive job. Adjust the parameters based on your needs. See [[Running jobs]] for more information.
To start the application, submit an interactive job. Adjust the parameters based on your needs. See [[Running jobs]] for more information.


</translate>
</translate>
{{Command|salloc --time{{=}}1:0:0 --ntasks{{=}}1 --cpus-per-task{{=}}2 --mem-per-cpu{{=}}1024M --account{{=}}def-yourpi srun notebook.sh
{{Command2|prompt=(jupyter_py3)_[name@server ~]$|salloc --time{{=}}1:0:0 --ntasks{{=}}1 --cpus-per-task{{=}}2 --mem-per-cpu{{=}}1024M --account{{=}}def-yourpi srun $VIRTUAL_ENV/bin/notebook.sh
|result=
|result=
salloc: Granted job allocation 1422754
salloc: Granted job allocation 1422754
salloc: Waiting for resource configuration
salloc: Nodes cdr544 are ready for job
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.662 NotebookApp] 0 active kernels
[I 14:07:08.662 NotebookApp] 0 active kernels
Line 114: Line 153:
Copy/paste this URL into your browser when you connect for the first time,
Copy/paste this URL into your browser when you connect for the first time,
     to login with a token:
     to login with a token:
         http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca}}
         http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e3}}
<translate>
<translate>


== Connect to the Notebook == <!--T:18-->
== Connecting to Jupyter Notebook == <!--T:18-->


<!--T:20-->
<!--T:20-->
To access the notebook running on a compute node from your web browser, you will need to create an '''SSH tunnel''' between the cluster and your computer since the compute nodes are not directly accessible from the Internet.  
To access Jupyter Notebook running on a compute node from your web browser, you will need to create an [[SSH tunnelling|SSH tunnel]] between the cluster and your computer since the compute nodes are not directly accessible from the Internet.


=== From Linux or MacOS X === <!--T:39-->
=== From Linux or MacOS X === <!--T:39-->


<!--T:40-->
<!--T:40-->
On a Linux or MacOS X system we recommend using the Python package [https://sshuttle.readthedocs.io sshuttle].
On a Linux or MacOS X system, we recommend using the [https://sshuttle.readthedocs.io sshuttle] Python package.


<!--T:21-->
<!--T:21-->
On your computer, open a new terminal window, and run the following sshuttle command to create the tunnel
On your computer, open a new terminal window and run the following <code>sshuttle</code> command to create the tunnel.


<!--T:22-->
<!--T:22-->
{{Command
{{Command2
|prompt=[name@my_computer $]
|prompt=[name@my_computer ~]$
|sshuttle --dns -Nr userid@machine_name}}
|sshuttle --dns -Nr <username>@<cluster>.computecanada.ca}}
 
<!--T:76-->
In the preceding command substitute <code><username></code> by your username; and substitute <code><cluster></code> by the cluster you connected to launch your Jupyter Notebook.


<!--T:35-->
<!--T:35-->
Then copy and paste the provided URL into your browser. In the example above this would be
Then, copy and paste the provided URL into your browser. In the above example, this would be
<pre>
<pre>
  http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca
  http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e3
</pre>
</pre>


Line 144: Line 186:


<!--T:42-->
<!--T:42-->
An SSH tunnel can be created from Windows using [MobaXTerm] as follows. Open two sessions in MobaXTerm.  
An [[SSH tunnelling|SSH tunnel]] can be created from Windows using [[Connecting_with_MobaXTerm | MobaXTerm]] as follows. This will also work from any Unix system (MacOS, Linux, etc).


<!--T:43-->
<!--T:43-->
Session 1 should be a connection to a cluster. Follow the instructions in Start the Notebook above to create a Jupyter notebook.
<ol>
<li>Open a new Terminal tab in MobaXTerm (Session 1) and connect to a cluster. Then follow the instructions in section [[#Starting_Jupyter_Notebook| Starting Jupyter Notebook]]. At this point, you should have on your screen an URL with the following form.
<pre>
http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e3
      └────────────────┬───────────────────┘        └──────────┬───────────┘
                  hostname:port                              token
</pre>
<li>Open a second Terminal tab in MobaXTerm (Session 2). In the following command, substitute <code><hostname:port></code> by its corresponding value from the URL you obtained in Session 1 (refer to the previous figure); substitute <code><username></code> by your username; and substitute <code><cluster></code> by the cluster you connected to in Session 1. Run the command.
{{Command2
|prompt=[name@my_computer ~]$
|ssh -L 8888:<hostname:port> <username>@<cluster>.computecanada.ca}}
<li> Open your browser and go to  
<pre>
http://localhost:8888/?token=<token>
</pre>
Replace <code><token></code> with its value from Session 1.
</ol>
 
== Shutting down Jupyter Notebook == <!--T:36-->


<!--T:44-->
<!--T:37-->
Session 2 should be a local terminal. In it we will set up the SSH tunnel. Run the following command, substituting the node name from the URL you received in Session 1. Following the example shown under Start the Notebook above:
You can shut down the Jupyter Notebook server before the walltime limit by pressing Ctrl-C twice in the terminal that launched the interactive job.


<!--T:45-->
<!--T:49-->
{{Command
If you used MobaXterm to create a tunnel, press Ctrl-D in Session 2 to shut down the tunnel.
|prompt=[name@my_computer ]$
| ssh -L 8888:cdr544.int.cedar.computecanada.ca:8888 someuser@cedar.computecanada.ca}}


<!--T:46-->
== Adding kernels == <!--T:54-->
The above command means that you will do a local port forwarding (-L), then it says that we will forward our local port 8888 to <code>cdr544.int.cedar.computecanada.ca:8888</code>, the hostname that was given when we started Jupyter Notebook. Now open your browser and go to


<!--T:47-->
<!--T:55-->
<pre>
It is possible to add kernels for other programming languages or Python versions different than the one running the Jupyter Notebook. Refer to [http://jupyter-client.readthedocs.io/en/latest/kernels.html Making kernels for Jupyter] to learn more.
http://localhost:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca
</pre>


<!--T:48-->
<!--T:56-->
Replace the token in this example with the one given you by Jupyter in Session 1. You can also type <code>http://localhost:8888</code> and there will be a prompt asking you for the token, which you can then copy and paste.
The installation of a new kernel is done in two steps.
#Installation of the packages that will allow the language interpreter to communicate with Jupyter Notebook.  
#Creation of a file that will indicate to Jupyter Notebook how to initiate a communication channel with the language interpreter. This file is called a ''kernel spec file''.


== Shut down the Notebook == <!--T:36-->
<!--T:57-->
Each kernel spec file has to be created in its own subfolder inside a folder in your home directory with the following path <code> ~/.local/share/jupyter/kernels</code>. Jupyter Notebook does not create this folder, so the first step in all cases is to create it. You can use the following command.
{{Command2|mkdir -p  ~/.local/share/jupyter/kernels}}


<!--T:37-->
<!--T:58-->
To shut down the Notebook server before the walltime limit, in the terminal that launched the interactive job, press Ctrl-C two times.
In the following sections, we provide a few examples of the kernel installation procedure.


<!--T:49-->
=== Julia === <!--T:63-->
If you used MobaXterm to create a tunnel, press Ctrl-D in Session 2 to shut down the tunnel.
</translate>


== Adding kernels ==
<!--T:64-->
<ol>
<li>Load the [[Julia]] module. {{Command2|module load julia}}
<li>Activate the Jupyter Notebook virtual environment. {{Command2|source $HOME/jupyter_py3/bin/activate}}
<li>Install IJulia. {{Command2|prompt=(jupyter_py3)_[name@server ~]$|echo 'Pkg.add("IJulia")' {{!}} julia}}
</ol>


It is possible to add kernels for other programmming languages or Python versions different than the one running the Jupyter Notebook. Refer to [http://jupyter-client.readthedocs.io/en/latest/kernels.html|Making kernels for Jupyter] to know more. The installation of a new kernel is done in two steps. The first step is to install the packages that will allow the language interpreter to communicate with the Jupyter Notebook. The second step is to create a file that will indicate to Jupyter Notebook how to initiate a communication channel with the language interpreter. This file is called a ''kernel spec file''.
<!--T:65-->
For more information, see the [https://github.com/JuliaLang/IJulia.jl IJulia documentation].


The kernel spec files have to be created in a single folder in your home directory with the following path <code> ~/.local/share/jupyter/kernels</code>. Jupyter Notebook does not create this folder, so the first step in all cases is to create it. You can use the following command.
=== Python === <!--T:77-->
{{Command|mkdir -p  ~/.local/share/jupyter/kernels}}


In the following sections, we provide a few examples of kernel installation and their respective kernel spec files.
<!--T:78-->
<ol>
<li>Load the Python module. {{Command2|module load python/3.5}}
<li>Create a new Python virtual environment. {{Command2|virtualenv $HOME/jupyter_py3.5}}
<li>Activate your newly created Python virtual environment. {{Command2|source $HOME/jupyter_py3.5/bin/activate}}
<li>Install the <code>ipykernel</code> library. {{Command2|prompt=(jupyter_py3.5)_[name@server ~]$|pip install ipykernel}}
<li>Generate the kernel spec file.  Substitute <code><unique_name></code> by a name that will uniquely identify your kernel. {{Command2|prompt=(jupyter_py3.5)_[name@server ~]$|python -m ipykernel install --user --name <unique_name> --display-name "Python 3.5 Kernel"}}
<li>Deactivate the virtual environment. {{Command2|prompt=(jupyter_py3.5)_[name@server ~]$|deactivate}}
</ol>


=== Anaconda ===
<!--T:79-->
For more information, see the [http://ipython.readthedocs.io/en/stable/install/kernel_install.html ipykernel documentation].


'''Before installing an Anaconda kernel''', make you have read the documentation and installed [[Anaconda]].
=== R === <!--T:66-->


# Load Anaconda module {{Command|module load miniconda3}}
<!--T:67-->
# '''Optional''' : Activate a specific conda virtual environment {{Command|source activate <your env>}}
<ol>
# Install the <code>ipykernel</code> library {{Command|pip install ipykernel}}
<li>Load the R module. {{Command2|module load r}}
# Generate the kernel spec file {{Command|python -m ipykernel --name <unique identifier without white space> --display-name "My Anaconda 3 Kernel"}}
<li>Activate the Jupyter Notebook virtual environment. {{Command2|source $HOME/jupyter_py3/bin/activate}}
# '''Optional''' : Deactivate the environment {{Command|source deactivate}}
<li>Install the R kernel dependencies. {{Command2|prompt=(jupyter_py3)_[name@server ~]$|R -e "install.packages(c('crayon', 'pbdZMQ', 'devtools'), repos{{=}}'http://cran.us.r-project.org')"}}
<li>Install the R kernel. {{Command2|prompt=(jupyter_py3)_[name@server ~]$|R -e "devtools::install_github(paste0('IRkernel/', c('repr', 'IRdisplay', 'IRkernel')))"}}
<li>Install the R kernel spec file. {{Command2|prompt=(jupyter_py3)_[name@server ~]$|R -e "IRkernel::installspec()"}}
</ol>


For more information, refer to [http://ipython.readthedocs.io/en/stable/install/kernel_install.html ipykernel documentation]
<!--T:68-->
For more information, see the [https://irkernel.github.io/docs/ IRKernel documentation].


== References ==
== References == <!--T:69-->
</translate>

Latest revision as of 16:24, 23 October 2023

Other languages:


Advanced material

This page is for advanced users. Please see JupyterHub instead.



Introduction

"Project Jupyter is a non-profit, open-source project, born out of the IPython Project in 2014 as it evolved to support interactive data science and scientific computing across all programming languages."[1]

"The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text."[2]

You can run Jupyter Notebook on a compute node or on a login node (not recommended). Note that login nodes impose various user- and process-based limits, so applications running there may be killed if they consume too much CPU time or memory. To use a compute node you will have to submit a job requesting the number of CPUs (and optionally GPUs), the amount of memory, and the run time. Here, we give instructions to submit a Jupyter Notebook job.

Other information:

  • Since Jupyter Notebook is the older Jupyter interface, please consider installing JupyterLab instead.
  • If you are instead looking for a preconfigured Jupyter environment, please see the Jupyter page.

Installing Jupyter Notebook

These instructions install Jupyter Notebook with the pip command in a Python virtual environment in your home directory. The following instructions are for Python 3.6, but you can also install the application for a different version by loading a different Python module.

  1. Load the Python module. 
    [name@server ~]$ module load python/3.7
  2. Create a new Python virtual environment. 
    [name@server ~]$ virtualenv $HOME/jupyter_py3
  3. Activate your newly created Python virtual environment. 
    [name@server ~]$ source $HOME/jupyter_py3/bin/activate
  4. Install Jupyter Notebook in your new virtual environment. 
    (jupyter_py3) [name@server ~]$ pip install --no-index --upgrade pip
(jupyter_py3) [name@server ~]$ pip install --no-index jupyter
  5. In the virtual environment, create a wrapper script that launches Jupyter Notebook. 
    (jupyter_py3)_[name@server ~]$ echo -e '#!/bin/bash\nexport JUPYTER_RUNTIME_DIR=$SLURM_TMPDIR/jupyter\njupyter notebook --ip $(hostname -f) --no-browser' > $VIRTUAL_ENV/bin/notebook.sh
  6. Finally, make the script executable. 
    (jupyter_py3)_[name@server ~]$ chmod u+x $VIRTUAL_ENV/bin/notebook.sh

Installing extensions

Extensions allow you to add functionalities and modify the application’s user interface.

Jupyter Lmod

Jupyter Lmod is an extension that allows you to interact with environment modules before launching kernels. The extension uses the Lmod's Python interface to accomplish module-related tasks like loading, unloading, saving a collection, etc. 

(jupyter_py3)_[name@server ~]$ pip install jupyterlmod
(jupyter_py3)_[name@server ~]$ jupyter nbextension install --py jupyterlmod --sys-prefix
(jupyter_py3)_[name@server ~]$ jupyter nbextension enable --py jupyterlmod --sys-prefix
(jupyter_py3)_[name@server ~]$ jupyter serverextension enable --py jupyterlmod --sys-prefix

Proxy web services

nbserverproxy enables users to reach arbitrary web services running within their spawned Jupyter server. This is useful to access web services that are listening only on a port of the localhost like TensorBoard. 

(jupyter_py3)_[name@server ~]$ pip install nbserverproxy
(jupyter_py3)_[name@server ~]$ jupyter serverextension enable --py nbserverproxy --sys-prefix

Example

In Jupyter, a user starts a web service via 'Terminal' in the New dropdown list: 

[name@server ~]$ tensorboard --port=8008


The service is proxied off of /proxy/ at https://address.of.notebook.server/user/theuser/proxy/8008.

RStudio Launcher

Jupyter Notebook can start an RStudio session that uses Jupyter Notebook's token authentication system. RStudio Launcher adds an RStudio Session option to the Jupyter Notebook New dropdown list.

Note: the installation procedure below only works with the StdEnv/2016.4 and StdEnv/2018.3 software environments. 

(jupyter_py3)_[name@server ~]$ pip install nbserverproxy
(jupyter_py3)_[name@server ~]$ pip install https://github.com/jupyterhub/nbrsessionproxy/archive/v0.8.0.zip
(jupyter_py3)_[name@server ~]$ jupyter serverextension enable --py nbserverproxy --sys-prefix
(jupyter_py3)_[name@server ~]$ jupyter nbextension install --py nbrsessionproxy --sys-prefix
(jupyter_py3)_[name@server ~]$ jupyter nbextension enable --py nbrsessionproxy --sys-prefix
(jupyter_py3)_[name@server ~]$ jupyter serverextension enable --py nbrsessionproxy --sys-prefix


Activating the environment

Once you have installed Jupyter Notebook, you need only reload the Python module associated with your environment when you log into the cluster. 

[name@server ~]$ module load python/3.7


Then, activate the virtual environment in which you have installed Jupyter Notebook. 

[name@server ~]$ source $HOME/jupyter_py3/bin/activate


RStudio Server (optional)

To use RStudio Launcher, load the RStudio Server module. 

(jupyter_py3)_[name@server ~]$ module load rstudio-server


Starting Jupyter Notebook

To start the application, submit an interactive job. Adjust the parameters based on your needs. See Running jobs for more information. 

(jupyter_py3)_[name@server ~]$ salloc --time=1:0:0 --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1024M --account=def-yourpi srun $VIRTUAL_ENV/bin/notebook.sh
salloc: Granted job allocation 1422754
salloc: Waiting for resource configuration
salloc: Nodes cdr544 are ready for job
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.662 NotebookApp] 0 active kernels
[I 14:07:08.662 NotebookApp] The Jupyter Notebook is running at:
[I 14:07:08.663 NotebookApp] http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca
[I 14:07:08.663 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 14:07:08.669 NotebookApp]

Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e3


Connecting to Jupyter Notebook

To access Jupyter Notebook running on a compute node from your web browser, you will need to create an SSH tunnel between the cluster and your computer since the compute nodes are not directly accessible from the Internet.

From Linux or MacOS X

On a Linux or MacOS X system, we recommend using the sshuttle Python package.

On your computer, open a new terminal window and run the following sshuttle command to create the tunnel. 

[name@my_computer ~]$ sshuttle --dns -Nr <username>@<cluster>.computecanada.ca


In the preceding command substitute <username> by your username; and substitute <cluster> by the cluster you connected to launch your Jupyter Notebook.

Then, copy and paste the provided URL into your browser. In the above example, this would be 

 http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e3

From Windows

An SSH tunnel can be created from Windows using MobaXTerm as follows. This will also work from any Unix system (MacOS, Linux, etc).

  1. Open a new Terminal tab in MobaXTerm (Session 1) and connect to a cluster. Then follow the instructions in section Starting Jupyter Notebook. At this point, you should have on your screen an URL with the following form. 
    http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e3
       └────────────────┬───────────────────┘        └──────────┬───────────┘
                  hostname:port                               token
  2. Open a second Terminal tab in MobaXTerm (Session 2). In the following command, substitute <hostname:port> by its corresponding value from the URL you obtained in Session 1 (refer to the previous figure); substitute <username> by your username; and substitute <cluster> by the cluster you connected to in Session 1. Run the command. 
    [name@my_computer ~]$ ssh -L 8888:<hostname:port> <username>@<cluster>.computecanada.ca
  3. Open your browser and go to 
     http://localhost:8888/?token=<token>

    Replace <token> with its value from Session 1.

Shutting down Jupyter Notebook

You can shut down the Jupyter Notebook server before the walltime limit by pressing Ctrl-C twice in the terminal that launched the interactive job.

If you used MobaXterm to create a tunnel, press Ctrl-D in Session 2 to shut down the tunnel.

Adding kernels

It is possible to add kernels for other programming languages or Python versions different than the one running the Jupyter Notebook. Refer to Making kernels for Jupyter to learn more.

The installation of a new kernel is done in two steps.

  1. Installation of the packages that will allow the language interpreter to communicate with Jupyter Notebook.
  2. Creation of a file that will indicate to Jupyter Notebook how to initiate a communication channel with the language interpreter. This file is called a kernel spec file.

Each kernel spec file has to be created in its own subfolder inside a folder in your home directory with the following path ~/.local/share/jupyter/kernels. Jupyter Notebook does not create this folder, so the first step in all cases is to create it. You can use the following command. 

[name@server ~]$ mkdir -p  ~/.local/share/jupyter/kernels


In the following sections, we provide a few examples of the kernel installation procedure.

Julia

  1. Load the Julia module. 
    [name@server ~]$ module load julia
  2. Activate the Jupyter Notebook virtual environment. 
    [name@server ~]$ source $HOME/jupyter_py3/bin/activate
  3. Install IJulia. 
    (jupyter_py3)_[name@server ~]$ echo 'Pkg.add("IJulia")' | julia

For more information, see the IJulia documentation.

Python

  1. Load the Python module. 
    [name@server ~]$ module load python/3.5
  2. Create a new Python virtual environment. 
    [name@server ~]$ virtualenv $HOME/jupyter_py3.5
  3. Activate your newly created Python virtual environment. 
    [name@server ~]$ source $HOME/jupyter_py3.5/bin/activate
  4. Install the ipykernel library. 
    (jupyter_py3.5)_[name@server ~]$ pip install ipykernel
  5. Generate the kernel spec file. Substitute <unique_name> by a name that will uniquely identify your kernel. 
    (jupyter_py3.5)_[name@server ~]$ python -m ipykernel install --user --name <unique_name> --display-name "Python 3.5 Kernel"
  6. Deactivate the virtual environment. 
    (jupyter_py3.5)_[name@server ~]$ deactivate

For more information, see the ipykernel documentation.

R

  1. Load the R module. 
    [name@server ~]$ module load r
  2. Activate the Jupyter Notebook virtual environment. 
    [name@server ~]$ source $HOME/jupyter_py3/bin/activate
  3. Install the R kernel dependencies. 
    (jupyter_py3)_[name@server ~]$ R -e "install.packages(c('crayon', 'pbdZMQ', 'devtools'), repos='http://cran.us.r-project.org')"
  4. Install the R kernel. 
    (jupyter_py3)_[name@server ~]$ R -e "devtools::install_github(paste0('IRkernel/', c('repr', 'IRdisplay', 'IRkernel')))"
  5. Install the R kernel spec file. 
    (jupyter_py3)_[name@server ~]$ R -e "IRkernel::installspec()"

For more information, see the IRKernel documentation.

References

  1. ↑ http://jupyter.org/about.html
  2. ↑ http://www.jupyter.org/
Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=JupyterNotebook&oldid=145808"