JupyterNotebook: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
No edit summary
(Marked this version for translation)
Line 2: Line 2:


<translate>
<translate>
=Introduction=
=Introduction= <!--T:1-->


<!--T:2-->
Jupyter notebook comes in one Python model on Graham. You can get it working on the login node (not recommended), and the compute nodes (highly recommended). Note that Login nodes impose various user- and process-based limits, so notebooks running there may be killed if they consume significant cpu-time or memory. You will have to submit a job requesting the # of CPU (or even GPU), amount of memory and runtime. Here, we give the instructions to submit a Jupyter job.
Jupyter notebook comes in one Python model on Graham. You can get it working on the login node (not recommended), and the compute nodes (highly recommended). Note that Login nodes impose various user- and process-based limits, so notebooks running there may be killed if they consume significant cpu-time or memory. You will have to submit a job requesting the # of CPU (or even GPU), amount of memory and runtime. Here, we give the instructions to submit a Jupyter job.


= Installing Jupyter Notebook =
= Installing Jupyter Notebook = <!--T:3-->


<!--T:4-->
These instructions install Jupyter into your home directory. To install Jupyter we will use the <code>pip</code> command and install it into a [[Python#Creating_and_using_a_virtual_environment | Python virtual environments]]. The below instructions install for Python 3.5.2. The below instructions install for Python 3.5.2 but you can also install for Python 3.5.Y or 2.7.X by loading a different Python module.
These instructions install Jupyter into your home directory. To install Jupyter we will use the <code>pip</code> command and install it into a [[Python#Creating_and_using_a_virtual_environment | Python virtual environments]]. The below instructions install for Python 3.5.2. The below instructions install for Python 3.5.2 but you can also install for Python 3.5.Y or 2.7.X by loading a different Python module.


<!--T:5-->
Load the Python module:
Load the Python module:
{{Command|module load python/3.5.2}}
{{Command|module load python/3.5.2}}
Line 15: Line 18:
{{Command|virtualenv $HOME/jupyter_py3}}
{{Command|virtualenv $HOME/jupyter_py3}}


<!--T:6-->
Activate your newly created python virtual environment:
Activate your newly created python virtual environment:
{{Command|source $HOME/jupyter_py3/bin/activate}}
{{Command|source $HOME/jupyter_py3/bin/activate}}


<!--T:7-->
Install Jupyter into your newly created virtual environment:
Install Jupyter into your newly created virtual environment:
{{Command
{{Command
Line 23: Line 28:
|pip install jupyter}}
|pip install jupyter}}


<!--T:8-->
In your virtual environment, create a wrapper script that launches Jupyter notebook  
In your virtual environment, create a wrapper script that launches Jupyter notebook  
{{Command
{{Command
Line 29: Line 35:
}}
}}


<!--T:9-->
Finally, make the script executable
Finally, make the script executable
{{Command
{{Command
Line 35: Line 42:
}}
}}


== Installing Extensions ==
== Installing Extensions == <!--T:10-->


<!--T:11-->
Extensions allow you to add functionalities and modify the appearance of the Notebook application.  
Extensions allow you to add functionalities and modify the appearance of the Notebook application.  


=== Jupyter Lmod ===
=== Jupyter Lmod === <!--T:12-->


<!--T:13-->
[https://github.com/cmd-ntrf/jupyter-lmod| Jupyter Lmod] is an extension that allows user to interact with environment modules before launching kernels. The extension use Lmod's Python interface to accomplish module related task like loading, unloading, saving collection, etc.
[https://github.com/cmd-ntrf/jupyter-lmod| Jupyter Lmod] is an extension that allows user to interact with environment modules before launching kernels. The extension use Lmod's Python interface to accomplish module related task like loading, unloading, saving collection, etc.


<!--T:14-->
{{Commands
{{Commands
|prompt=(jupyter_py3)[name@server $]
|prompt=(jupyter_py3)[name@server $]
Line 51: Line 61:
}}
}}


=== RStudio Launcher ===
=== RStudio Launcher === <!--T:15-->


<!--T:16-->
Jupyter can be used to start an RStudio session that use Jupyter token authentication system. This extension add an "RStudio Session" button to the New notebook menu.
Jupyter can be used to start an RStudio session that use Jupyter token authentication system. This extension add an "RStudio Session" button to the New notebook menu.


<!--T:17-->
{{Commands
{{Commands
|prompt=(jupyter_py3)[name@server $]
|prompt=(jupyter_py3)[name@server $]
Line 65: Line 77:
}}
}}


= Connecting to a manually spawned Jupyter Notebook =
= Connecting to a manually spawned Jupyter Notebook = <!--T:18-->


== Create a Tunnel ==
== Create a Tunnel == <!--T:19-->


<!--T:20-->
To access the notebook running on a compute node from your web browser, you will need to create a tunnel between the cluster and your computer since the compute nodes are not directly accessible from the Internet. To create that tunnel, we recommend the usage of the Python package [https://sshuttle.readthedocs.io sshuttle].
To access the notebook running on a compute node from your web browser, you will need to create a tunnel between the cluster and your computer since the compute nodes are not directly accessible from the Internet. To create that tunnel, we recommend the usage of the Python package [https://sshuttle.readthedocs.io sshuttle].


<!--T:21-->
On your computer, open a new terminal window, and run the following sshuttle command to create the tunnel
On your computer, open a new terminal window, and run the following sshuttle command to create the tunnel


<!--T:22-->
{{Command
{{Command
|prompt=[name@my_computer $]
|prompt=[name@my_computer $]
|sshuttle --dns -Nr userid@machine_name}}
|sshuttle --dns -Nr userid@machine_name}}


== Activate the Environment ==
== Activate the Environment == <!--T:23-->


<!--T:24-->
On the cluster, load the Python module associated with your environment:
On the cluster, load the Python module associated with your environment:


<!--T:25-->
{{Command|module load python/3.5.2}}
{{Command|module load python/3.5.2}}


<!--T:26-->
Then, activate the virtual environment in which you have installed Jupyter:
Then, activate the virtual environment in which you have installed Jupyter:


<!--T:27-->
{{Command|source $HOME/jupyter_py3/bin/activate}}
{{Command|source $HOME/jupyter_py3/bin/activate}}


=== RStudio Server (optional) ===  
=== RStudio Server (optional) === <!--T:28-->


<!--T:29-->
If you have installed the RStudio launcher extension and wish to use it, you will have to load the RStudio Server module.
If you have installed the RStudio launcher extension and wish to use it, you will have to load the RStudio Server module.


<!--T:30-->
{{Command|module load rstudio-server}}
{{Command|module load rstudio-server}}


== Start the Notebook ==
== Start the Notebook == <!--T:31-->


<!--T:32-->
To start the Notebook, submit an interactive job. Adjust the parameters based on your needs. See [[Running jobs]] for more information.
To start the Notebook, submit an interactive job. Adjust the parameters based on your needs. See [[Running jobs]] for more information.
{{Command|salloc --time{{=}}1:0:0 --ntasks{{=}}1 --cpus-per-task{{=}}2 --mem-per-cpu{{=}}1024M --account{{=}}def-yourpi srun notebook.sh
{{Command|salloc --time{{=}}1:0:0 --ntasks{{=}}1 --cpus-per-task{{=}}2 --mem-per-cpu{{=}}1024M --account{{=}}def-yourpi srun notebook.sh
|result=
|result=


<!--T:33-->
salloc: Granted job allocation 1422754
salloc: Granted job allocation 1422754
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
Line 107: Line 130:
[C 14:07:08.669 NotebookApp]
[C 14:07:08.669 NotebookApp]


     Copy/paste this URL into your browser when you connect for the first time,
     <!--T:34-->
Copy/paste this URL into your browser when you connect for the first time,
     to login with a token:
     to login with a token:
         http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca}}
         http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca}}


<!--T:35-->
Copy/paste the provided URL into your browser and enjoy your notebook.
Copy/paste the provided URL into your browser and enjoy your notebook.


== Shutdown the Notebook ==
== Shutdown the Notebook == <!--T:36-->


<!--T:37-->
To shutdown the Notebook server before the walltime limit, in the terminal that launched the interactive job, press Ctrl-C two times.
To shutdown the Notebook server before the walltime limit, in the terminal that launched the interactive job, press Ctrl-C two times.
</translate>
</translate>

Revision as of 13:25, 6 October 2017

Other languages:

Introduction

Jupyter notebook comes in one Python model on Graham. You can get it working on the login node (not recommended), and the compute nodes (highly recommended). Note that Login nodes impose various user- and process-based limits, so notebooks running there may be killed if they consume significant cpu-time or memory. You will have to submit a job requesting the # of CPU (or even GPU), amount of memory and runtime. Here, we give the instructions to submit a Jupyter job.

Installing Jupyter Notebook

These instructions install Jupyter into your home directory. To install Jupyter we will use the pip command and install it into a Python virtual environments. The below instructions install for Python 3.5.2. The below instructions install for Python 3.5.2 but you can also install for Python 3.5.Y or 2.7.X by loading a different Python module.

Load the Python module:

Question.png 
[name@server ~]$ module load python/3.5.2

Create a new python virtual environment:

Question.png 
[name@server ~]$ virtualenv $HOME/jupyter_py3

Activate your newly created python virtual environment:

Question.png 
[name@server ~]$ source $HOME/jupyter_py3/bin/activate

Install Jupyter into your newly created virtual environment:

Question.png 
(jupyter_py3)[name@server $] pip install jupyter

In your virtual environment, create a wrapper script that launches Jupyter notebook

Question.png 
(jupyter_py3)[name@server $] echo -e '#!/bin/bash\nunset XDG_RUNTIME_DIR\njupyter notebook --ip $(hostname -f) --no-browser' > $VIRTUAL_ENV/bin/notebook.sh

Finally, make the script executable

Question.png 
(jupyter_py3)[name@server $] chmod u+x $VIRTUAL_ENV/bin/notebook.sh

Installing Extensions

Extensions allow you to add functionalities and modify the appearance of the Notebook application.

Jupyter Lmod

Jupyter Lmod is an extension that allows user to interact with environment modules before launching kernels. The extension use Lmod's Python interface to accomplish module related task like loading, unloading, saving collection, etc. 

(jupyter_py3)[name@server $] pip install jupyterlmod
(jupyter_py3)[name@server $] jupyter nbextension install --py jupyterlmod --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension enable --py jupyterlmod --sys-prefix
(jupyter_py3)[name@server $] jupyter serverextension enable --py jupyterlmod --sys-prefix


RStudio Launcher

Jupyter can be used to start an RStudio session that use Jupyter token authentication system. This extension add an "RStudio Session" button to the New notebook menu. 

(jupyter_py3)[name@server $] pip install nbserverproxy
(jupyter_py3)[name@server $] pip install git+https://github.com/cmd-ntrf/nbrsessionproxy
(jupyter_py3)[name@server $] jupyter serverextension enable --py nbserverproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension install --py nbrsessionproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter nbextension enable --py nbrsessionproxy --sys-prefix
(jupyter_py3)[name@server $] jupyter serverextension enable --py nbrsessionproxy --sys-prefix


Connecting to a manually spawned Jupyter Notebook

Create a Tunnel

To access the notebook running on a compute node from your web browser, you will need to create a tunnel between the cluster and your computer since the compute nodes are not directly accessible from the Internet. To create that tunnel, we recommend the usage of the Python package sshuttle.

On your computer, open a new terminal window, and run the following sshuttle command to create the tunnel

Question.png 
[name@my_computer $] sshuttle --dns -Nr userid@machine_name

Activate the Environment

On the cluster, load the Python module associated with your environment:

Question.png 
[name@server ~]$ module load python/3.5.2

Then, activate the virtual environment in which you have installed Jupyter:

Question.png 
[name@server ~]$ source $HOME/jupyter_py3/bin/activate

RStudio Server (optional)

If you have installed the RStudio launcher extension and wish to use it, you will have to load the RStudio Server module.

Question.png 
[name@server ~]$ module load rstudio-server

Start the Notebook

To start the Notebook, submit an interactive job. Adjust the parameters based on your needs. See Running jobs for more information.

Question.png 
[name@server ~]$ salloc --time=1:0:0 --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1024M --account=def-yourpi srun notebook.sh
salloc: Granted job allocation 1422754
[I 14:07:08.661 NotebookApp] Serving notebooks from local directory: /home/fafor10
[I 14:07:08.662 NotebookApp] 0 active kernels
[I 14:07:08.662 NotebookApp] The Jupyter Notebook is running at:
[I 14:07:08.663 NotebookApp] http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca
[I 14:07:08.663 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 14:07:08.669 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://cdr544.int.cedar.computecanada.ca:8888/?token=7ed7059fad64446f837567e32af8d20efa72e72476eb72ca

Copy/paste the provided URL into your browser and enjoy your notebook.

Shutdown the Notebook

To shutdown the Notebook server before the walltime limit, in the terminal that launched the interactive job, press Ctrl-C two times.

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=JupyterNotebook&oldid=39276"