QGIS: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
(added stuff and I don't even know how I created the page in the first place)
(added QT_QPA_PLATFORM=offscreen env to batch job; seems required to avoid graphical output depenencies)
 
(21 intermediate revisions by one other user not shown)
Line 1: Line 1:
<languages/>
<languages/>
[[Category:Software]]
[[Category:Software]]
{{Draft}}
<translate>
<translate>


Line 7: Line 8:


<!--T:2-->
<!--T:2-->
!!IMPORTANT!!
IMPORTANT
 
<!--T:3-->
Never make intense use of QGIS on the login nodes! Submit jobs using the command line whenever possible and if you must visualize your data using the GUI, please do so on an interactive node. Using parallel rendering on the shared login nodes will result in the termination of your session.
Never make intense use of QGIS on the login nodes! Submit jobs using the command line whenever possible and if you must visualize your data using the GUI, please do so on an interactive node. Using parallel rendering on the shared login nodes will result in the termination of your session.


== Loading QGIS ==  
== Loading QGIS == <!--T:4-->
<!--T:3-->
You will first need to load gcc
You will first need to load gcc
{{Command
{{Command
Line 17: Line 19:
}}  
}}  


<!--T:4-->
<!--T:5-->
Then, you will need to load the QGIS module; there could potentially be several versions available and you can see a list of all of them using the command  
Then, you will need to load the QGIS module; there could potentially be several versions available and you can see a list of all of them using the command  
{{Command
{{Command
Line 27: Line 29:
}}  
}}  


<!--T:5-->
<!--T:6-->
You might also have to load various other modules depending on the packages you need to install. For example, "rgdal" will require that you load a module called "gdal", which itself requires that you load nixpkgs and gcc. Nixpkgs should already be loaded by default. You can ensure that it is by running
You might also have to load various other modules depending on the packages you need to install. For example, "rgdal" will require that you load a module called "gdal", which itself requires that you load nixpkgs and gcc. Nixpkgs should already be loaded by default. You can ensure that it is by running


<!--T:6-->
<!--T:7-->
{{Command
{{Command
|module list
|module list
}}  
}}  


<!--T:7-->
<!--T:8-->
If nixpkgs is not listed, you can load it by running
If nixpkgs is not listed, you can load it by running


<!--T:8-->
<!--T:9-->
{{Command
{{Command
|module load nixpkgs/16.09
|module load nixpkgs/16.09
}}  
}}  


<!--T:9-->
<!--T:10-->
If any package fails to install, be sure to read the error message carefully, as it might give you some details concerning some additional modules you need to load. You can also find out if a module is dependent on any other module by running
If any package fails to install, be sure to read the error message carefully, as it might give you some details concerning some additional modules you need to load. You can also find out if a module is dependent on any other module by running
{{Command
{{Command
Line 50: Line 52:




====Command-line QGIS==== <!--T:31-->
== Command-line QGIS == <!--T:11-->
You can execute QGIS by running a Python script:
You can execute QGIS by running a Python script:


<!--T:16-->
<!--T:12-->
{{Command|python qgis_code.py}}
{{Command|python qgis_code.py}}


<!--T:32-->
<!--T:13-->
1. Place your python code in a script file, in this case the [https://docs.qgis.org/testing/en/docs/pyqgis_developer_cookbook/intro.html file] is called ''qgis_code.py''.
1. Place your python code in a script file, in this case the [https://docs.qgis.org/testing/en/docs/pyqgis_developer_cookbook/intro.html file] is called ''qgis_code.py''.


<!--T:33-->
<!--T:14-->
{{File
{{File
  |name=qgis_code.py
  |lang="python"
  |contents=
from qgis.core import *
from qgis.core import *


Line 82: Line 87:
}}
}}


<!--T:34-->
<!--T:14-->
2. Copy the following content in a job submission script called ''job.sh'':
2. Copy the following content in a job submission script called ''job.sh'':


  <!--T:35-->
  <!--T:15-->
{{File
{{File
   |name=job.sh
   |name=job.sh
Line 98: Line 103:
module load qgis
module load qgis


export QT_QPA_PLATFORM=offscreen
python qgis_code.py
python qgis_code.py
}}
}}


<!--T:36-->
<!--T:16-->
3. Submit the job with:
3. Submit the job with:


  <!--T:37-->
  <!--T:17-->
<source lang="bash">
<source lang="bash">
sbatch job.sh
sbatch job.sh
</source>
</source>


<!--T:38-->
<!--T:18-->
For more on submitting jobs, see the [[Running jobs]] page.
For more on submitting jobs, see the [[Running jobs]] page.


== Using the GUI == <!--T:19-->
You may want to visualize large datasets without having to download them to your desktop. QGIS offers the possibility to use multiple cores to render the maps but you should not use this option on the login nodes which are shared with multiple users. Instead, you should use interactive nodes to do the visualization.


==Using the GUI== <!--T:46-->
<!--T:20-->
 
Using the QGIS GUI requires X11 forwarding, which you should make sure is enabled.
 
 
 
====Running==== <!--T:31-->
 
<!--T:32-->
1. Place your R code in a script file, in this case the file is called ''test.R''.
 
<!--T:33-->
{{File
  |name=test.R
  |lang="r"
  |contents=
#Tell all slaves to return a message identifying themselves.
library("Rmpi")
sprintf("TEST mpi.universe.size() =  %i", mpi.universe.size())
ns <- mpi.universe.size() - 1
sprintf("TEST attempt to spawn %i slaves", ns)
mpi.spawn.Rslaves(nslaves=ns)
mpi.remote.exec(paste("I am",mpi.comm.rank(),"of",mpi.comm.size()))
mpi.remote.exec(paste(mpi.comm.get.parent()))
#Send execution commands to the slaves
x<-5
#These would all be pretty correlated one would think
x<-mpi.remote.exec(rnorm,x)
length(x)
x
mpi.close.Rslaves()
mpi.quit()
}}
 
<!--T:34-->
2. Copy the following content in a job submission script called ''job.sh'':
 
<!--T:35-->
{{File
  |name=job.sh
  |lang="bash"
  |contents=
#!/bin/bash
#SBATCH --account=def-someacct  # replace this with your own account
#SBATCH --ntasks=5              # number of MPI processes
#SBATCH --mem-per-cpu=2048M      # memory; default unit is megabytes
#SBATCH --time=0-00:15          # time (DD-HH:MM)
module load r/3.4.0
module load openmpi/1.10.7
export R_LIBS=~/local/R_libs/
mpirun -np 1 R CMD BATCH test.R test.txt
}}
 
<!--T:36-->
3. Submit the job with:


<!--T:37-->
==== MobaXTerm (Windows) ====<!--T:21-->
<source lang="bash">
You can find information on how to connect with MobaXTerm and use X11 fowarding on the [[Connecting_with_MobaXTerm]] page.
sbatch job.sh
</source>


<!--T:38-->
<!--T:22-->
For more on submitting jobs, see the [[Running jobs]] page.
1. In the main window, click "Sessions" and a new window will pop up. Select "SSH".


=== doParallel and foreach === <!--T:39-->
2. In "Remote host", write the name of the server you plan to use, for example "cedar.computecanada.ca".  
====Usage====
Foreach can be considered as a unified interface for all backends (i.e. doMC, doMPI, doParallel, doRedis, etc.). It works on all platforms, assuming that the backend works. doParallel acts as an interface between foreach and the parallel package and can be loaded alone. There are some known efficiency issues when using foreach to run a very large number of very small tasks. Therefore, keep in mind that the following code is not the best example of an optimized use of the foreach() call but rather that the function chosen was kept at a minimum for demonstration purposes.


<!--T:48-->
3. You can choose to specify a username and write your Compute Canada username. Leave port 22 as it is.
You must register the backend by feeding it the number of cores available. If the backend is not registered, foreach will assume that the number of cores is 1 and will proceed to go through the iterations serially.  
[[File:Mobaxterm ssh2.png|720px|frame|left]]
<br clear=all> <!-- This is to prevent the next section from filling to the right of the image. -->


The general method to use foreach is:
4. A terminal will open and you will be prompted to enter your password. You won't see anything appear as you input your password but it's nothing to worry about.
# to load both foreach and the backend package;
# to register the backend;
# to call foreach() by keeping it on the same line as the %do% (serial) or %dopar% operator.
====Running==== <!--T:40-->


<!--T:49-->
5. Make sure that the X server logo is green and that X11 forwarding is therefore enabled.
1. Place your R code in a script file, in this case the file is called ''test_foreach.R''.
[[File:Mobaxterm x11.png|720px|frame|left]]
<br clear=all>
6. In the terminal, load gcc and the QGIS module as shown in the section "Loading QGIS".


<!--T:41-->
7. Start QGIS by typing
{{File
{{Command|qgis}}
  |name=test_foreach.R
  |lang="r"
  |contents=
# library(foreach) # optional if using doParallel
library(doParallel) #


<!--T:50-->
8. QGIS should appear in a new window and you are ready to use it.
# a very simple function
[[File:Qgis window small.png|720px|frame|left]]
test_func <- function(var1, var2) {
<br clear=all> <!-- This is to prevent the next section from filling to the right of the image. -->
    return(var1*var2)
}


<!--T:51-->
9. To close QGIS, simply close the window like you usually would.  
# we will iterate over two sets of values, you can modify this to explore the mechanism of foreach
var1.v = c(1:8)
var2.v = seq(0.1, 1, length.out = 8)


<!--T:52-->
==== Terminal (Unix) ====<!--T:22-->
# Use the environment variable SLURM_NTASKS to set the number of cores.
More information to come.
# This is for SLURM. Replace SLURM_NTASKS by the proper variable for your system.
# Avoid manually setting a number of cores.
ncores = Sys.getenv("SLURM_NTASKS")


<!--T:53-->
==== Use an interactive node ====<!--T:23-->
registerDoParallel(cores=ncores)# Shows the number of Parallel Workers to be used
Runtime is limited on the login nodes, so you will need to request a testing job to have more time for exploring and visualizing your data.
print(ncores) # this how many cores are available, and how many you have requested.
Additionally, by doing so, you will have access to the 40 cores of each of the nodes requested.
getDoParWorkers()# you can compare with the number of actual workers


<!--T:54-->
<!--T:24-->
# be careful! foreach() and %dopar% must be on the same line!
request an interactive job, ie.
foreach(var1=var1.v, .combine=rbind) %:% foreach(var2=var2.v, .combine=rbind) %dopar% {test_func(var1=var1, var2=var2)}
  [name@login ~]$ salloc --time=1:0:0 --ntasks=2 --x11 --account=def-someuser
}}


<!--T:42-->
this will connect you to a node. Note the --x11 argument, which will enable X11 forwarding on the interactive node.  
2. Copy the following content in a job submission script called ''job_foreach.sh'':


<!--T:43-->
<!--T:25-->
{{File
You can then load QGIS and run it on the interactive node.
  |name=job_foreach.sh
{{Command|module load gcc}}
  |lang="bash"
{{Command|module load qgis}}
  |contents=
{{Command|qgis}}
#!/bin/bash
#SBATCH --account=def-someacct  # replace this with your own account
#SBATCH --ntasks=4              # number of processes
#SBATCH --mem-per-cpu=2048M      # memory; default unit is megabytes
#SBATCH --time=0-00:15          # time (DD-HH:MM)
#SBATCH --mail-user=yourname@someplace.com # Send email updates to you or someone else
#SBATCH --mail-type=ALL          # send an email in all cases (job started, job ended, job aborted)


<!--T:55-->
<!--T:26-->
module load r/3.4.3
You can enable parallel rendering of the layers by clicking "Settings" and "Options" in the main window, then "Rendering" in the Options window. Check "Render layers in parallel using many CPU cores" and you can select a number of cores to use (optional).
export R_LIBS=~/local/R_libs/
[[File:Parallel rendering2.png|720px|frame|left]]
R CMD BATCH --no-save --no-restore test_foreach.R
}}


<!--T:44-->
<br clear=all> <!-- This is to prevent the next section from filling to the right of the image. -->
3. Submit the job with:


<!--T:45-->
{{Command|sbatch job_foreach.sh}}


<!--T:47-->
<!--T:27-->
For more on submitting jobs, see the [[Running jobs]] page.
Once you are done and have closed QGIS, you can terminate the allocation
{{Command|exit}}


</translate>
</translate>

Latest revision as of 21:47, 13 May 2024


This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.




QGIS is a free and open-source cross-platform desktop geographic information system (GIS) application that supports viewing, editing, and analysis of geospatial data.

IMPORTANT

Never make intense use of QGIS on the login nodes! Submit jobs using the command line whenever possible and if you must visualize your data using the GUI, please do so on an interactive node. Using parallel rendering on the shared login nodes will result in the termination of your session.

Loading QGIS[edit]

You will first need to load gcc

Question.png 
[name@server ~]$ module load gcc/5.4.0

Then, you will need to load the QGIS module; there could potentially be several versions available and you can see a list of all of them using the command

Question.png 
[name@server ~]$ module spider qgis

You can load a particular QGIS module using a command like

Question.png 
[name@server ~]$ module load qgis/2.18.24

You might also have to load various other modules depending on the packages you need to install. For example, "rgdal" will require that you load a module called "gdal", which itself requires that you load nixpkgs and gcc. Nixpkgs should already be loaded by default. You can ensure that it is by running

Question.png 
[name@server ~]$ module list

If nixpkgs is not listed, you can load it by running

Question.png 
[name@server ~]$ module load nixpkgs/16.09

If any package fails to install, be sure to read the error message carefully, as it might give you some details concerning some additional modules you need to load. You can also find out if a module is dependent on any other module by running

Question.png 
[name@server ~]$ module spider gdal/2.2.1


Command-line QGIS[edit]

You can execute QGIS by running a Python script:

Question.png 
[name@server ~]$ python qgis_code.py

1. Place your python code in a script file, in this case the file is called qgis_code.py.


File : qgis_code.py

from qgis.core import *

# supply path to qgis install location
QgsApplication.setPrefixPath("/path/to/qgis/installation", True)

# create a reference to the QgsApplication, setting the
# second argument to False disables the GUI
qgs = QgsApplication([], False)

# load providers
qgs.initQgis()

# Write your code here to load some layers, use processing
# algorithms, etc.

# When your script is complete, call exitQgis() to remove the
# provider and layer registries from memory

qgs.exitQgis()


2. Copy the following content in a job submission script called job.sh:


File : job.sh

#!/bin/bash
#SBATCH --account=def-someacct   # replace this with your own account
#SBATCH --ntasks=1               # number of MPI processes
#SBATCH --mem-per-cpu=2048M      # memory; default unit is megabytes
#SBATCH --time=0-00:15           # time (DD-HH:MM)
module load gcc
module load qgis

export QT_QPA_PLATFORM=offscreen
python qgis_code.py


3. Submit the job with: 

sbatch job.sh

For more on submitting jobs, see the Running jobs page.

Using the GUI[edit]

You may want to visualize large datasets without having to download them to your desktop. QGIS offers the possibility to use multiple cores to render the maps but you should not use this option on the login nodes which are shared with multiple users. Instead, you should use interactive nodes to do the visualization.

Using the QGIS GUI requires X11 forwarding, which you should make sure is enabled.

MobaXTerm (Windows)[edit]

You can find information on how to connect with MobaXTerm and use X11 fowarding on the Connecting_with_MobaXTerm page.

1. In the main window, click "Sessions" and a new window will pop up. Select "SSH".

2. In "Remote host", write the name of the server you plan to use, for example "cedar.computecanada.ca".

3. You can choose to specify a username and write your Compute Canada username. Leave port 22 as it is.

Mobaxterm ssh2.png


4. A terminal will open and you will be prompted to enter your password. You won't see anything appear as you input your password but it's nothing to worry about.

5. Make sure that the X server logo is green and that X11 forwarding is therefore enabled.

Mobaxterm x11.png


6. In the terminal, load gcc and the QGIS module as shown in the section "Loading QGIS".

7. Start QGIS by typing

Question.png 
[name@server ~]$ qgis

8. QGIS should appear in a new window and you are ready to use it.

Qgis window small.png


9. To close QGIS, simply close the window like you usually would.

Terminal (Unix)[edit]

More information to come.

Use an interactive node[edit]

Runtime is limited on the login nodes, so you will need to request a testing job to have more time for exploring and visualizing your data. Additionally, by doing so, you will have access to the 40 cores of each of the nodes requested.

request an interactive job, ie. 

 [name@login ~]$ salloc --time=1:0:0 --ntasks=2 --x11 --account=def-someuser

this will connect you to a node. Note the --x11 argument, which will enable X11 forwarding on the interactive node.

You can then load QGIS and run it on the interactive node.

Question.png 
[name@server ~]$ module load gcc
Question.png 
[name@server ~]$ module load qgis
Question.png 
[name@server ~]$ qgis

You can enable parallel rendering of the layers by clicking "Settings" and "Options" in the main window, then "Rendering" in the Options window. Check "Render layers in parallel using many CPU cores" and you can select a number of cores to use (optional).

Parallel rendering2.png



Once you are done and have closed QGIS, you can terminate the allocation

Question.png 
[name@server ~]$ exit

Message Passing Interface

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=QGIS&oldid=153667"