JupyterNotebook: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
No edit summary
mNo edit summary
 
(144 intermediate revisions by 15 users not shown)
Line 1: Line 1:
<languages />
<languages />
{{Warning|title=Advanced material|content=This page is for advanced users. Please see [[JupyterHub]] instead.}}


<translate>
<translate>
=Introduction= <!--T:1-->
==Introduction== <!--T:1-->


<!--T:38-->
<!--T:38-->
[http://jupyter.org/about.html Project Jupyter] is an open source project, born out of the [https://ipython.org/ IPython Project], as it evolved to support interactive data science and scientific computing across all programming languages. The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text <ref>http://www.jupyter.org/, The Jupyter Notebook</ref>.
"Project Jupyter is a non-profit, open-source project, born out of the IPython Project in 2014 as it evolved to support interactive data science and scientific computing across all programming languages."<ref>http://jupyter.org/about.html</ref>
 
<!--T:51-->
"The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text."<ref>http://www.jupyter.org/</ref>


<!--T:2-->
<!--T:2-->
You can run Jupyter Notebook on the login node (not recommended) or the compute nodes (highly recommended). Note that login nodes impose various user- and process-based limits, so notebooks running there may be killed if they consume significant cpu-time or memory. To use a compute node you will have to submit a job requesting the number of CPUs (and, optionally, GPUs), the amount of memory, and the run time. Here, we give the instructions to submit a Jupyter Notebook job.
You can run Jupyter Notebook on a compute node or on a login node (not recommended). Note that login nodes impose various user- and process-based limits, so applications running there may be killed if they consume too much CPU time or memory. To use a compute node you will have to submit a job requesting the number of CPUs (and optionally GPUs), the amount of memory, and the run time. Here, we give instructions to submit a Jupyter Notebook job.


= Installing Jupyter Notebook = <!--T:3-->
<!--T:50-->
'''Other information:'''
* Since Jupyter Notebook is the older Jupyter interface, please consider installing '''[[Advanced_Jupyter_configuration|JupyterLab]]''' instead.
* If you are instead looking for a preconfigured Jupyter environment, please see the '''[[Jupyter]]''' page.
 
== Installing Jupyter Notebook == <!--T:3-->


<!--T:4-->
<!--T:4-->
These instructions install Jupyter Notebook with the <code>pip</code> command in a [[Python#Creating_and_using_a_virtual_environment | Python virtual environment]] in your home directory. The following instructions are for Python 3.5.2, but you can also install the application for a different version by loading a different Python module.
These instructions install Jupyter Notebook with the <code>pip</code> command in a [[Python#Creating_and_using_a_virtual_environment | Python virtual environment]] in your home directory. The following instructions are for Python 3.6, but you can also install the application for a different version by loading a different Python module.


<!--T:5-->
<!--T:5-->
#Load the Python module:
<ol>
#:{{Command|module load python/3.5.2}}
<li>Load the Python module.
#Create a new Python virtual environment:
{{Command2|module load python/3.7}}
#:{{Command|virtualenv $HOME/jupyter_py3}}
<li>Create a new Python virtual environment.
#Activate your newly created Python virtual environment:
{{Command2|virtualenv $HOME/jupyter_py3}}
#:{{Command|source $HOME/jupyter_py3/bin/activate}}
<li>Activate your newly created Python virtual environment.
#Install Jupyter into your virtual environment:
{{Command2|source $HOME/jupyter_py3/bin/activate}}
#:{{Command
<li>Install Jupyter Notebook in your new virtual environment.
|prompt=(jupyter_py3)[name@server $]
{{Commands2
|pip install jupyter}}
|prompt=(jupyter_py3) [name@server ~]$
#In your virtual environment, create a wrapper script that launches Jupyter notebook
|pip install --no-index --upgrade pip
#:{{Command
|pip install --no-index jupyter
|prompt=(jupyter_py3)[name@server $]
|echo -e '#!/bin/bash\nunset XDG_RUNTIME_DIR\njupyter notebook --ip $(hostname -f) --no-browser' > $VIRTUAL_ENV/bin/notebook.sh
}}
}}
#Finally, make the script executable
<li>In the virtual environment, create a wrapper script that launches Jupyter Notebook.
#:{{Command
{{Command2
|prompt=(jupyter_py3)[name@server $]
|prompt=(jupyter_py3)_[name@server ~]$
|echo -e '#!/bin/bash\nexport JUPYTER_RUNTIME_DIR{{=}}$SLURM_TMPDIR/jupyter\njupyter notebook --ip $(hostname -f) --no-browser' > $VIRTUAL_ENV/bin/notebook.sh
}}
<li>Finally, make the script executable.
{{Command2
|prompt=(jupyter_py3)_[name@server ~]$
|chmod u+x $VIRTUAL_ENV/bin/notebook.sh
|chmod u+x $VIRTUAL_ENV/bin/notebook.sh
}}
}}
</ol>


== Installing Extensions == <!--T:10-->
== Installing extensions == <!--T:10-->


<!--T:11-->
<!--T:11-->
Extensions allow you to add functionalities and modify the appearance of the Notebook application.  
Extensions allow you to add functionalities and modify the application’s user interface.  


=== Jupyter Lmod === <!--T:12-->
=== Jupyter Lmod === <!--T:12-->


<!--T:13-->
<!--T:13-->
[https://github.com/cmd-ntrf/jupyter-lmod| Jupyter Lmod] is an extension that allows you to interact with environment modules before launching kernels. The extension use Lmod's Python interface to accomplish module-related tasks like loading, unloading, saving a collection, etc.
[https://github.com/cmd-ntrf/jupyter-lmod Jupyter Lmod] is an extension that allows you to interact with environment modules before launching kernels. The extension uses the Lmod's Python interface to accomplish module-related tasks like loading, unloading, saving a collection, etc.


<!--T:14-->
{{Commands2
{{Commands
|prompt=(jupyter_py3)_[name@server ~]$
|prompt=(jupyter_py3)[name@server $]
|pip install jupyterlmod
|pip install jupyterlmod
|jupyter nbextension install --py jupyterlmod --sys-prefix
|jupyter nbextension install --py jupyterlmod --sys-prefix
Line 55: Line 68:
|jupyter serverextension enable --py jupyterlmod --sys-prefix
|jupyter serverextension enable --py jupyterlmod --sys-prefix
}}
}}
=== Proxy web services === <!--T:14-->
<!--T:75-->
[https://github.com/jupyterhub/nbserverproxy nbserverproxy] enables users to reach arbitrary web services running within their spawned Jupyter server. This is useful to access web services that are listening only on a port of the localhost like [https://www.tensorflow.org/programmers_guide/summaries_and_tensorboard TensorBoard].
{{Commands2
|prompt=(jupyter_py3)_[name@server ~]$
|pip install nbserverproxy
|jupyter serverextension enable --py nbserverproxy --sys-prefix
}}
==== Example ==== <!--T:71-->
<!--T:72-->
In Jupyter, a user starts a web service via 'Terminal' in the ''New'' dropdown list:
<!--T:73-->
{{Command2
|tensorboard --port{{=}}8008
}}
<!--T:74-->
The service is proxied off of /proxy/ at https://address.of.notebook.server/user/theuser/proxy/8008.


=== RStudio Launcher === <!--T:15-->
=== RStudio Launcher === <!--T:15-->


<!--T:16-->
<!--T:16-->
Jupyter can start an RStudio session that uses Jupyter's token authentication system. This extension adds an "RStudio Session" button to the New notebook menu.
Jupyter Notebook can start an RStudio session that uses Jupyter Notebook's token authentication system. RStudio Launcher adds an ''RStudio Session'' option to the Jupyter Notebook ''New'' dropdown list.
 
<!--T:81-->
'''Note:''' the installation procedure below only works with the <code>StdEnv/2016.4</code> and <code>StdEnv/2018.3</code> software environments.


<!--T:17-->
<!--T:17-->
{{Commands
{{Commands2
|prompt=(jupyter_py3)[name@server $]
|prompt=(jupyter_py3)_[name@server ~]$
|pip install nbserverproxy  
|pip install nbserverproxy  
|pip install git+https://github.com/cmd-ntrf/nbrsessionproxy
|pip install https://github.com/jupyterhub/nbrsessionproxy/archive/v0.8.0.zip
|jupyter serverextension enable --py nbserverproxy --sys-prefix
|jupyter serverextension enable --py nbserverproxy --sys-prefix
|jupyter nbextension install --py nbrsessionproxy --sys-prefix
|jupyter nbextension install --py nbrsessionproxy --sys-prefix
Line 72: Line 110:
}}
}}


= Connecting to a manually spawned Jupyter Notebook = <!--T:18-->
== Activating the environment == <!--T:23-->
 
== Create a tunnel == <!--T:19-->
 
<!--T:20-->
To access the notebook running on a compute node from your web browser, you will need to create a tunnel between the cluster and your computer since the compute nodes are not directly accessible from the Internet.
 
=== Linux or MacOS X ===
 
On a Linux or MacOS X system we recommend using the Python package [https://sshuttle.readthedocs.io sshuttle].
 
<!--T:21-->
On your computer, open a new terminal window, and run the following sshuttle command to create the tunnel
 
<!--T:22-->
{{Command
|prompt=[name@my_computer $]
|sshuttle --dns -Nr userid@machine_name}}
 
=== Windows ===
 
We have tested MobaXTerm on Windows 10 using ssh as follows. Open 2 sessions, (1) will have the above instructions, and (2) will have the ssh port-forwarding.
 
Session 1: when you run the salloc, you will get something like this,
 
<pre>
(jupyter_py3) [jnandez@gra-login1 ~]$ salloc --time=1:0:0 --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1024M --account=cc-debug srun $VIRTUAL_ENV/bin/notebook.sh
salloc: Pending job allocation 1060408
salloc: job 1060408 queued and waiting for resources
salloc: job 1060408 has been allocated resources
salloc: Granted job allocation 1060408
[I 12:53:31.208 NotebookApp] Loading lmod extension
[I 12:53:31.213 NotebookApp] Serving notebooks from local directory: /home/jnandez
[I 12:53:31.213 NotebookApp] 0 active kernels
[I 12:53:31.213 NotebookApp] The Jupyter Notebook is running at:
[I 12:53:31.213 NotebookApp] http://gra800.graham.sharcnet:8888/?token=9f31fae47604d65cf7706e706960a2f519b437d22f85eca1
[I 12:53:31.213 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 12:53:31.216 NotebookApp]
 
    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://gra800.graham.sharcnet:8888/?token=9f31fae47604d65cf7706e706960a2f519b437d22f85eca1
</pre>
 
Session 2: run the following command
 
{{Command
|prompt=[name@my_computer ]$
| ssh -L 8888:gra800.graham.sharcnet:8888 jnandez@graham.computecanada.ca}}
 
The above command means  that you will do a local port forwarding (-L), then it says that we will forward our local port 8888 to gra800.graham.sharcnet:8888. Note that you can pick another local port, we have used the same so we know that we are working remotely. Now open your browser and go to
 
http://localhost:8888/?token=9f31fae47604d65cf7706e706960a2f519b437d22f85eca1
 
The token is the same as printed by Session 1. You can also type http://localhost:8888, and there will be a prompt asking you for the token, which you will copy and paste from Session 1.
 
== Activate the environment == <!--T:23-->


<!--T:24-->
<!--T:24-->
On the cluster, load the Python module associated with your environment:
Once you have installed Jupyter Notebook, you need only reload the Python module associated with your environment when you log into the cluster.


<!--T:25-->
<!--T:25-->
{{Command|module load python/3.5.2}}
{{Command2|module load python/3.7}}


<!--T:26-->
<!--T:26-->
Then, activate the virtual environment in which you have installed Jupyter:
Then, activate the virtual environment in which you have installed Jupyter Notebook.


<!--T:27-->
<!--T:27-->
{{Command|source $HOME/jupyter_py3/bin/activate}}
{{Command2|source $HOME/jupyter_py3/bin/activate}}


=== RStudio Server (optional) === <!--T:28-->  
=== RStudio Server (optional) === <!--T:28-->  


<!--T:29-->
<!--T:29-->
If you have installed the RStudio launcher extension and wish to use it, you will have to load the RStudio Server module.
To use [[#RStudio_Launcher | RStudio Launcher]], load the RStudio Server module.


<!--T:30-->
<!--T:30-->
{{Command|module load rstudio-server}}
{{Command2|prompt=(jupyter_py3)_[name@server ~]$
|module load rstudio-server}}


== Start the Notebook == <!--T:31-->
== Starting Jupyter Notebook == <!--T:31-->


<!--T:32-->
<!--T:32-->
To start the Notebook, submit an interactive job. Adjust the parameters based on your needs. See [[Running jobs]] for more information.
To start the application, submit an interactive job. Adjust the parameters based on your needs. See [[Running jobs]] for more information.


</translate>
</translate>
{{Command|salloc --time{{=}}1:0:0 --ntasks{{=}}1 --cpus-per-task{{=}}2 --mem-per-cpu{{=}}1024M --account{{=}}def-yourpi srun notebook.sh
{{Command2|prompt=(jupyter_py3)_[name@server ~]$|salloc --time{{=}}1:0:0 --ntasks{{=}}1 --cpus-per-task{{=}}2 --mem-per-cpu{{=}}1024M --account{{=}}def-yourpi srun $VIRTUAL_ENV/bin/notebook.sh
|result=
|result=
salloc: Granted job allocation 1422754
salloc: Granted job allocation 1422754
salloc: Waiting for resource configuration
salloc: Nodes cdr544 are ready for job
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.662 NotebookApp] 0 active kernels
[I 14:07:08.662 NotebookApp] 0 active kernels
Line 169: Line 153:
Copy/paste this URL into your browser when you connect for the first time,
Copy/paste this URL into your browser when you connect for the first time,
     to login with a token:
     to login with a token:
         http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca}}
         http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e3}}
<translate>
 
== Connecting to Jupyter Notebook == <!--T:18-->
 
<!--T:20-->
To access Jupyter Notebook running on a compute node from your web browser, you will need to create an [[SSH tunnelling|SSH tunnel]] between the cluster and your computer since the compute nodes are not directly accessible from the Internet.
 
=== From Linux or MacOS X === <!--T:39-->
 
<!--T:40-->
On a Linux or MacOS X system, we recommend using the [https://sshuttle.readthedocs.io sshuttle] Python package.
 
<!--T:21-->
On your computer, open a new terminal window and run the following <code>sshuttle</code> command to create the tunnel.
 
<!--T:22-->
{{Command2
|prompt=[name@my_computer ~]$
|sshuttle --dns -Nr <username>@<cluster>.computecanada.ca}}
 
<!--T:76-->
In the preceding command substitute <code><username></code> by your username; and substitute <code><cluster></code> by the cluster you connected to launch your Jupyter Notebook.


<translate>
<!--T:35-->
<!--T:35-->
Copy/paste the provided URL into your browser and enjoy your notebook.
Then, copy and paste the provided URL into your browser. In the above example, this would be
<pre>
http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e3
</pre>
 
=== From Windows === <!--T:41-->
 
<!--T:42-->
An [[SSH tunnelling|SSH tunnel]] can be created from Windows using [[Connecting_with_MobaXTerm | MobaXTerm]] as follows.  This will also work from any Unix system (MacOS, Linux, etc).
 
<!--T:43-->
<ol>
<li>Open a new Terminal tab in MobaXTerm (Session 1) and connect to a cluster. Then follow the instructions in section [[#Starting_Jupyter_Notebook| Starting Jupyter Notebook]]. At this point, you should have on your screen an URL with the following form.
<pre>
http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e3
      └────────────────┬───────────────────┘        └──────────┬───────────┘
                  hostname:port                              token
</pre>
<li>Open a second Terminal tab in MobaXTerm (Session 2). In the following command, substitute <code><hostname:port></code> by its corresponding value from the URL you obtained in Session 1 (refer to the previous figure); substitute <code><username></code> by your username; and substitute <code><cluster></code> by the cluster you connected to in Session 1. Run the command.
{{Command2
|prompt=[name@my_computer ~]$
|ssh -L 8888:<hostname:port> <username>@<cluster>.computecanada.ca}}
<li> Open your browser and go to
<pre>
http://localhost:8888/?token=<token>
</pre>
Replace <code><token></code> with its value from Session 1.
</ol>


== Shut down the Notebook == <!--T:36-->
== Shutting down Jupyter Notebook == <!--T:36-->


<!--T:37-->
<!--T:37-->
To shut down the Notebook server before the walltime limit, in the terminal that launched the interactive job, press Ctrl-C two times.
You can shut down the Jupyter Notebook server before the walltime limit by pressing Ctrl-C twice in the terminal that launched the interactive job.
 
<!--T:49-->
If you used MobaXterm to create a tunnel, press Ctrl-D in Session 2 to shut down the tunnel.
 
== Adding kernels == <!--T:54-->
 
<!--T:55-->
It is possible to add kernels for other programming languages or Python versions different than the one running the Jupyter Notebook. Refer to [http://jupyter-client.readthedocs.io/en/latest/kernels.html Making kernels for Jupyter] to learn more.
 
<!--T:56-->
The installation of a new kernel is done in two steps.
#Installation of the packages that will allow the language interpreter to communicate with Jupyter Notebook.
#Creation of a file that will indicate to Jupyter Notebook how to initiate a communication channel with the language interpreter. This file is called a ''kernel spec file''.
 
<!--T:57-->
Each kernel spec file has to be created in its own subfolder inside a folder in your home directory with the following path <code> ~/.local/share/jupyter/kernels</code>. Jupyter Notebook does not create this folder, so the first step in all cases is to create it. You can use the following command.
{{Command2|mkdir -p  ~/.local/share/jupyter/kernels}}
 
<!--T:58-->
In the following sections, we provide a few examples of the kernel installation procedure.
 
=== Julia === <!--T:63-->
 
<!--T:64-->
<ol>
<li>Load the [[Julia]] module. {{Command2|module load julia}}
<li>Activate the Jupyter Notebook virtual environment. {{Command2|source $HOME/jupyter_py3/bin/activate}}
<li>Install IJulia. {{Command2|prompt=(jupyter_py3)_[name@server ~]$|echo 'Pkg.add("IJulia")' {{!}} julia}}
</ol>
 
<!--T:65-->
For more information, see the [https://github.com/JuliaLang/IJulia.jl IJulia documentation].
 
=== Python === <!--T:77-->
 
<!--T:78-->
<ol>
<li>Load the Python module. {{Command2|module load python/3.5}}
<li>Create a new Python virtual environment. {{Command2|virtualenv $HOME/jupyter_py3.5}}
<li>Activate your newly created Python virtual environment. {{Command2|source $HOME/jupyter_py3.5/bin/activate}}
<li>Install the <code>ipykernel</code> library. {{Command2|prompt=(jupyter_py3.5)_[name@server ~]$|pip install ipykernel}}
<li>Generate the kernel spec file.  Substitute <code><unique_name></code> by a name that will uniquely identify your kernel. {{Command2|prompt=(jupyter_py3.5)_[name@server ~]$|python -m ipykernel install --user --name <unique_name> --display-name "Python 3.5 Kernel"}}
<li>Deactivate the virtual environment. {{Command2|prompt=(jupyter_py3.5)_[name@server ~]$|deactivate}}
</ol>
 
<!--T:79-->
For more information, see the [http://ipython.readthedocs.io/en/stable/install/kernel_install.html ipykernel documentation].
 
=== R === <!--T:66-->
 
<!--T:67-->
<ol>
<li>Load the R module. {{Command2|module load r}}
<li>Activate the Jupyter Notebook virtual environment. {{Command2|source $HOME/jupyter_py3/bin/activate}}
<li>Install the R kernel dependencies. {{Command2|prompt=(jupyter_py3)_[name@server ~]$|R -e "install.packages(c('crayon', 'pbdZMQ', 'devtools'), repos{{=}}'http://cran.us.r-project.org')"}}
<li>Install the R kernel. {{Command2|prompt=(jupyter_py3)_[name@server ~]$|R -e "devtools::install_github(paste0('IRkernel/', c('repr', 'IRdisplay', 'IRkernel')))"}}
<li>Install the R kernel spec file. {{Command2|prompt=(jupyter_py3)_[name@server ~]$|R -e "IRkernel::installspec()"}}
</ol>
 
<!--T:68-->
For more information, see the [https://irkernel.github.io/docs/ IRKernel documentation].
 
== References == <!--T:69-->
</translate>
</translate>
== References ==

Latest revision as of 16:24, 23 October 2023

Other languages:


Advanced material

This page is for advanced users. Please see JupyterHub instead.



Introduction

"Project Jupyter is a non-profit, open-source project, born out of the IPython Project in 2014 as it evolved to support interactive data science and scientific computing across all programming languages."[1]

"The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text."[2]

You can run Jupyter Notebook on a compute node or on a login node (not recommended). Note that login nodes impose various user- and process-based limits, so applications running there may be killed if they consume too much CPU time or memory. To use a compute node you will have to submit a job requesting the number of CPUs (and optionally GPUs), the amount of memory, and the run time. Here, we give instructions to submit a Jupyter Notebook job.

Other information:

  • Since Jupyter Notebook is the older Jupyter interface, please consider installing JupyterLab instead.
  • If you are instead looking for a preconfigured Jupyter environment, please see the Jupyter page.

Installing Jupyter Notebook

These instructions install Jupyter Notebook with the pip command in a Python virtual environment in your home directory. The following instructions are for Python 3.6, but you can also install the application for a different version by loading a different Python module.

  1. Load the Python module. 
    [name@server ~]$ module load python/3.7
  2. Create a new Python virtual environment. 
    [name@server ~]$ virtualenv $HOME/jupyter_py3
  3. Activate your newly created Python virtual environment. 
    [name@server ~]$ source $HOME/jupyter_py3/bin/activate
  4. Install Jupyter Notebook in your new virtual environment. 
    (jupyter_py3) [name@server ~]$ pip install --no-index --upgrade pip
(jupyter_py3) [name@server ~]$ pip install --no-index jupyter
  5. In the virtual environment, create a wrapper script that launches Jupyter Notebook. 
    (jupyter_py3)_[name@server ~]$ echo -e '#!/bin/bash\nexport JUPYTER_RUNTIME_DIR=$SLURM_TMPDIR/jupyter\njupyter notebook --ip $(hostname -f) --no-browser' > $VIRTUAL_ENV/bin/notebook.sh
  6. Finally, make the script executable. 
    (jupyter_py3)_[name@server ~]$ chmod u+x $VIRTUAL_ENV/bin/notebook.sh

Installing extensions

Extensions allow you to add functionalities and modify the application’s user interface.

Jupyter Lmod

Jupyter Lmod is an extension that allows you to interact with environment modules before launching kernels. The extension uses the Lmod's Python interface to accomplish module-related tasks like loading, unloading, saving a collection, etc. 

(jupyter_py3)_[name@server ~]$ pip install jupyterlmod
(jupyter_py3)_[name@server ~]$ jupyter nbextension install --py jupyterlmod --sys-prefix
(jupyter_py3)_[name@server ~]$ jupyter nbextension enable --py jupyterlmod --sys-prefix
(jupyter_py3)_[name@server ~]$ jupyter serverextension enable --py jupyterlmod --sys-prefix

Proxy web services

nbserverproxy enables users to reach arbitrary web services running within their spawned Jupyter server. This is useful to access web services that are listening only on a port of the localhost like TensorBoard. 

(jupyter_py3)_[name@server ~]$ pip install nbserverproxy
(jupyter_py3)_[name@server ~]$ jupyter serverextension enable --py nbserverproxy --sys-prefix

Example

In Jupyter, a user starts a web service via 'Terminal' in the New dropdown list: 

[name@server ~]$ tensorboard --port=8008


The service is proxied off of /proxy/ at https://address.of.notebook.server/user/theuser/proxy/8008.

RStudio Launcher

Jupyter Notebook can start an RStudio session that uses Jupyter Notebook's token authentication system. RStudio Launcher adds an RStudio Session option to the Jupyter Notebook New dropdown list.

Note: the installation procedure below only works with the StdEnv/2016.4 and StdEnv/2018.3 software environments. 

(jupyter_py3)_[name@server ~]$ pip install nbserverproxy
(jupyter_py3)_[name@server ~]$ pip install https://github.com/jupyterhub/nbrsessionproxy/archive/v0.8.0.zip
(jupyter_py3)_[name@server ~]$ jupyter serverextension enable --py nbserverproxy --sys-prefix
(jupyter_py3)_[name@server ~]$ jupyter nbextension install --py nbrsessionproxy --sys-prefix
(jupyter_py3)_[name@server ~]$ jupyter nbextension enable --py nbrsessionproxy --sys-prefix
(jupyter_py3)_[name@server ~]$ jupyter serverextension enable --py nbrsessionproxy --sys-prefix


Activating the environment

Once you have installed Jupyter Notebook, you need only reload the Python module associated with your environment when you log into the cluster. 

[name@server ~]$ module load python/3.7


Then, activate the virtual environment in which you have installed Jupyter Notebook. 

[name@server ~]$ source $HOME/jupyter_py3/bin/activate


RStudio Server (optional)

To use RStudio Launcher, load the RStudio Server module. 

(jupyter_py3)_[name@server ~]$ module load rstudio-server


Starting Jupyter Notebook

To start the application, submit an interactive job. Adjust the parameters based on your needs. See Running jobs for more information. 

(jupyter_py3)_[name@server ~]$ salloc --time=1:0:0 --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1024M --account=def-yourpi srun $VIRTUAL_ENV/bin/notebook.sh
salloc: Granted job allocation 1422754
salloc: Waiting for resource configuration
salloc: Nodes cdr544 are ready for job
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.662 NotebookApp] 0 active kernels
[I 14:07:08.662 NotebookApp] The Jupyter Notebook is running at:
[I 14:07:08.663 NotebookApp] http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca
[I 14:07:08.663 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 14:07:08.669 NotebookApp]

Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e3


Connecting to Jupyter Notebook

To access Jupyter Notebook running on a compute node from your web browser, you will need to create an SSH tunnel between the cluster and your computer since the compute nodes are not directly accessible from the Internet.

From Linux or MacOS X

On a Linux or MacOS X system, we recommend using the sshuttle Python package.

On your computer, open a new terminal window and run the following sshuttle command to create the tunnel. 

[name@my_computer ~]$ sshuttle --dns -Nr <username>@<cluster>.computecanada.ca


In the preceding command substitute <username> by your username; and substitute <cluster> by the cluster you connected to launch your Jupyter Notebook.

Then, copy and paste the provided URL into your browser. In the above example, this would be 

 http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e3

From Windows

An SSH tunnel can be created from Windows using MobaXTerm as follows. This will also work from any Unix system (MacOS, Linux, etc).

  1. Open a new Terminal tab in MobaXTerm (Session 1) and connect to a cluster. Then follow the instructions in section Starting Jupyter Notebook. At this point, you should have on your screen an URL with the following form. 
    http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e3
       └────────────────┬───────────────────┘        └──────────┬───────────┘
                  hostname:port                               token
  2. Open a second Terminal tab in MobaXTerm (Session 2). In the following command, substitute <hostname:port> by its corresponding value from the URL you obtained in Session 1 (refer to the previous figure); substitute <username> by your username; and substitute <cluster> by the cluster you connected to in Session 1. Run the command. 
    [name@my_computer ~]$ ssh -L 8888:<hostname:port> <username>@<cluster>.computecanada.ca
  3. Open your browser and go to 
     http://localhost:8888/?token=<token>

    Replace <token> with its value from Session 1.

Shutting down Jupyter Notebook

You can shut down the Jupyter Notebook server before the walltime limit by pressing Ctrl-C twice in the terminal that launched the interactive job.

If you used MobaXterm to create a tunnel, press Ctrl-D in Session 2 to shut down the tunnel.

Adding kernels

It is possible to add kernels for other programming languages or Python versions different than the one running the Jupyter Notebook. Refer to Making kernels for Jupyter to learn more.

The installation of a new kernel is done in two steps.

  1. Installation of the packages that will allow the language interpreter to communicate with Jupyter Notebook.
  2. Creation of a file that will indicate to Jupyter Notebook how to initiate a communication channel with the language interpreter. This file is called a kernel spec file.

Each kernel spec file has to be created in its own subfolder inside a folder in your home directory with the following path ~/.local/share/jupyter/kernels. Jupyter Notebook does not create this folder, so the first step in all cases is to create it. You can use the following command. 

[name@server ~]$ mkdir -p  ~/.local/share/jupyter/kernels


In the following sections, we provide a few examples of the kernel installation procedure.

Julia

  1. Load the Julia module. 
    [name@server ~]$ module load julia
  2. Activate the Jupyter Notebook virtual environment. 
    [name@server ~]$ source $HOME/jupyter_py3/bin/activate
  3. Install IJulia. 
    (jupyter_py3)_[name@server ~]$ echo 'Pkg.add("IJulia")' | julia

For more information, see the IJulia documentation.

Python

  1. Load the Python module. 
    [name@server ~]$ module load python/3.5
  2. Create a new Python virtual environment. 
    [name@server ~]$ virtualenv $HOME/jupyter_py3.5
  3. Activate your newly created Python virtual environment. 
    [name@server ~]$ source $HOME/jupyter_py3.5/bin/activate
  4. Install the ipykernel library. 
    (jupyter_py3.5)_[name@server ~]$ pip install ipykernel
  5. Generate the kernel spec file. Substitute <unique_name> by a name that will uniquely identify your kernel. 
    (jupyter_py3.5)_[name@server ~]$ python -m ipykernel install --user --name <unique_name> --display-name "Python 3.5 Kernel"
  6. Deactivate the virtual environment. 
    (jupyter_py3.5)_[name@server ~]$ deactivate

For more information, see the ipykernel documentation.

R

  1. Load the R module. 
    [name@server ~]$ module load r
  2. Activate the Jupyter Notebook virtual environment. 
    [name@server ~]$ source $HOME/jupyter_py3/bin/activate
  3. Install the R kernel dependencies. 
    (jupyter_py3)_[name@server ~]$ R -e "install.packages(c('crayon', 'pbdZMQ', 'devtools'), repos='http://cran.us.r-project.org')"
  4. Install the R kernel. 
    (jupyter_py3)_[name@server ~]$ R -e "devtools::install_github(paste0('IRkernel/', c('repr', 'IRdisplay', 'IRkernel')))"
  5. Install the R kernel spec file. 
    (jupyter_py3)_[name@server ~]$ R -e "IRkernel::installspec()"

For more information, see the IRKernel documentation.

References

  1. ↑ http://jupyter.org/about.html
  2. ↑ http://www.jupyter.org/
Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=JupyterNotebook&oldid=145808"