cc_staff
107
edits
GATK: Difference between revisions
no edit summary
mNo edit summary |
No edit summary |
||
| Line 47: | Line 47: | ||
==General usage == | ==General usage == | ||
The later versions of GATK (>=4.0.0.0) provide a wrapper over the | The later versions of GATK (>=4.0.0.0) provide a wrapper over the java executables (.jar). Loading the GATK modules will automatically set most of the environmental variables you will need to successfully run GATK. | ||
java executables (.jar). Loading the GATK modules will automatically set | |||
most of the environmental variables you will need to successfully run GATK. | |||
The module spider command also provides you with usage and examples of that | The module spider command also provides you with usage and examples of that wrapper: | ||
wrapper: | |||
<pre> | <pre> | ||
Usage | Usage | ||
| Line 64: | Line 61: | ||
</pre> | </pre> | ||
As you probably notice, there are some arguments to be passed directly to java through the '''--java-options''' such as the maximum heap memory ( | As you probably notice, there are some arguments to be passed directly to java through the '''--java-options''' such as the maximum heap memory (<code>-Xmx8G</code> in the example, reserving 8 Gb of memory for the virtual machine). We recommend that you '''always''' use <code>-DGATK_STACKTRACE_ON_USER_EXCEPTION=true</code> since it will give you more information in case the program fails. This information can help you or us (in case you needed support) to solve the issue. | ||
Note that all options passed to | Note that all options passed to <code>--java-options</code> have to be within quotation marks. | ||
===Earlier versions than GATK 4 === | ===Earlier versions than GATK 4 === | ||
| Line 74: | Line 71: | ||
</pre> | </pre> | ||
However, GenomeAnalysisTK.jar must be in PATH. In Compute Canada systems, the environmental variables | However, GenomeAnalysisTK.jar must be in PATH. In Compute Canada systems, the environmental variables <code>$EBROOTPICARD</code> for Picard (included in GATK >= 4) and <code>$EBROOTGATK</code> for GATK contain the path to the jar file, so the appropriate way to call GATK <= 3 is: | ||
<pre> | <pre> | ||
module load gatk/3.8 | module load gatk/3.8 | ||
java -jar $EBROOTGATK/GenomeAnalysisTK.jar PROGRAM OPTIONS | java -jar "${EBROOTGATK}"/GenomeAnalysisTK.jar PROGRAM OPTIONS | ||
</pre> | </pre> | ||
You can find the specific usage of GATK <= 3 in the [https://gatkforums.broadinstitute.org/gatk/categories/gatk-guide GATK3 guide]. | You can find the specific usage of GATK <= 3 in the [https://gatkforums.broadinstitute.org/gatk/categories/gatk-guide GATK3 guide]. | ||
===Multicore usage === | |||
Most GATK (>=4) tools are not multicore by default. This means that you should request only one core when calling these kind of tools. Some tools use threads in some of the computations (e.g. <code>Mutect2</code> has the <code>--native-pair-hmm-threads</code>) and therefore you can require more cpus (most of them with up to 4 threads) for these computations. GATK4, however, does provides '''some''' [https://gatk.broadinstitute.org/hc/en-us/articles/360035890591-Spark SPARK commands]: | |||
<pre> | |||
Not all GATK tools use Spark | |||
Tools that can use Spark generally have a note to that effect in their respective Tool Doc. | |||
- Some GATK tools exist in distinct Spark-capable and non-Spark-capable versions | |||
The "sparkified" versions have the suffix "Spark" at the end of their names. Many of these are still experimental; down the road we plan to consolidate them so that there will be only one version per tool. | |||
- Some GATK tools only exist in a Spark-capable version | |||
Those tools don't have the "Spark" suffix. | |||
</pre> | |||
For the commands that do use spark, you can request multiple cpus. '''NOTE:''' please provide the exact number of cpus to the spark command, so if you requested 10 cpus, use <code>--spark-master local[10]</code> instead of <code>--spark-master local[*]</code>. If you want to scale the spark calls with multinode SPARK cluster, you have to first [https://docs.computecanada.ca/wiki/Apache_Spark/en deploy an SPARK cluster] and then set up the appropriate variables in the GATK4 spark command. | |||
==Frequently asked questions == | ==Frequently asked questions == | ||
| Line 97: | Line 110: | ||
--CREATE_INDEX true | --CREATE_INDEX true | ||
</pre> | </pre> | ||
This assumes that your input file is sorted by coordinates and will generate an index along with the annotated output (--CREATE_INDEX true) | This assumes that your input file is sorted by coordinates and will generate an index along with the annotated output (<code>--CREATE_INDEX true</code>) | ||
===How do I deal with java.lang.OutOfMemoryError: Java heap space === | ===How do I deal with <code>java.lang.OutOfMemoryError: Java heap space</code> === | ||
Oftentimes the subprograms of GATK require more memory to process your files. If you were not using the | Oftentimes the subprograms of GATK require more memory to process your files. If you were not using the <code>-Xms</code> command, add it to the <code>--java-options</code>. For example, let's imagine that you run the following command: | ||
<pre> | <pre> | ||
gatk MarkDuplicates \ | gatk MarkDuplicates \ | ||
| Line 108: | Line 121: | ||
</pre> | </pre> | ||
But it gives you the | But it gives you the <code>java.lang.OutOfMemoryError: Java heap space</code> error. Try: | ||
<pre> | <pre> | ||
| Line 118: | Line 131: | ||
</pre> | </pre> | ||
If it fails again, keep increasing the memory until you find the required memory for your particular data set. If you are using any of our systems, remember to request enough memory for this. | If it fails again, keep increasing the memory until you find the required memory for your particular data set. If you are using any of our systems, '''remember to request enough memory for this'''. | ||
If you are interested in knowing more about java heap space, you can start [https://plumbr.io/outofmemoryerror/java-heap-space here]. | If you are interested in knowing more about java heap space, you can start [https://plumbr.io/outofmemoryerror/java-heap-space here]. | ||
===Increasing the heap memory does not fix the java.lang.OutOfMemoryError: Java heap space === | ===Increasing the heap memory does not fix the <code>java.lang.OutOfMemoryError: Java heap space</code> === | ||
There are cases in which the memory issue cannot be fixed with increasing the heap memory. This often happens with non-model organisms, and you are using too many scaffolds in your reference. In this case it is recommended to remove small scaffolds and create subsets of your reference. This implies that you have to map multiple times and run the pipelines in each of the subsets. '''This approach does not work in all pipelines''' so review your results carefully. GATK is designed with the human genome in mind, and therefore other organism will require adjustment in many parameters and pipelines. | There are cases in which the memory issue cannot be fixed with increasing the heap memory. This often happens with non-model organisms, and you are using too many scaffolds in your reference. In this case it is recommended to remove small scaffolds and create subsets of your reference. This implies that you have to map multiple times and run the pipelines in each of the subsets. '''This approach does not work in all pipelines''' so review your results carefully. GATK is designed with the human genome in mind, and therefore other organism will require adjustment in many parameters and pipelines. | ||
===Using more resources than asked for === | ===Using more resources than asked for === | ||
Sometimes GATK/JAVA applications will use more memory or | Sometimes GATK/JAVA applications will use more memory or CPUs/threads than the ones requested. This is often generated by the JAVA garbage collection. To add control for this, you can add <code>-XX:ConcGCThreads=1</code> to the <code>--java-options</code> argument. | ||
===FAQ on GATK === | ===FAQ on GATK === | ||
| Line 132: | Line 145: | ||
=References = | =References = | ||
[https://gatk.broadinstitute.org/hc/en-us GATK Home] | [https://gatk.broadinstitute.org/hc/en-us GATK Home] | ||
[https://gatk.broadinstitute.org/hc/en-us/articles/360035532012-Parallelism-Multithreading-Scatter-Gather GATK SPARK] | |||
[https://gatk.broadinstitute.org/hc/en-us/articles/360035889611-How-can-I-make-GATK-tools-run-faster- Make GATK run faster] | |||
[[Category:Bioinformatics]] | [[Category:Bioinformatics]] | ||
[[Category:Software]] | [[Category:Software]] | ||