Julia: Difference between revisions

Jump to navigation Jump to search

Julia (view source)

Revision as of 18:35, 7 September 2021

2,516 bytes removed ,  3 years ago
Undo revision 103156 by Bcardoen (talk)
(Undo revision 103159 by Mboisson (talk))
Tag: Undo
(Undo revision 103156 by Bcardoen (talk))
Tag: Undo
Line 7: Line 7:


<!--T:2-->
<!--T:2-->
[https://julialang.org Julia] is a programming language that was designed from the beginning for performance, ease of use and portability. It is available as a [[Utiliser_des_modules/en | module]] on Compute Canada clusters.
[https://julialang.org Julia] is a programming language that was designed from the beginning for performance, ease of use and portability. It is is available as a [[Utiliser_des_modules/en | module]] on Compute Canada clusters.


= Compiling packages = <!--T:3-->
= Compiling packages = <!--T:3-->
Line 32: Line 32:


<!--T:8-->
<!--T:8-->
In the example above, installing just the JLD package creates a <code>~/.julia</code> tree with 18673 files and directories and using 236M of space, almost 5% of a standard user's quota for <code>/home</code>.  It's worth remembering that installing a lot of packages will consume a lot of space.  
In the example above, installing just the JLD package creates a <code>~/.julia</code> tree with 18673 files and directories and using 236M of space, almost 5% of a standard user's quota for <code>/home</code>.  It's worth remembering that installing a lot of packages will consume a lot of space.
 
In addition, because all Julia's user files are saved in $HOME/.julia, significant slowdowns can be experienced when working with non-trivial dependencies. When one uses Julia's distributed programming mode, this overhead is increased due to k Julia processes accessing $HOME/.julia/compiled. On interactive (salloc) or sbatch nodes, one can opt for a workaround by:
{{Command
|export JULIA_DEPOT_PATH="$SLURM_TMPDIR:$JULIA_DEPOT_PATH"
}}
 
With $SLURM_TMPDIR residing on a very fast local disk, installation, updates, precompilation, and reads to registries are much faster (up to x100). To make Julia only use this temporary path, one can use the following workaround
{{File
  |name=julia_install.jl
  |lang="julia"
  |contents=
import Pkg
## Create local env in current dir (before you add packages)
## If current dir has an existing (local) env
# Pkg.activate(".")
## Else
Pkg.instantiate()
 
## Add your application(s), julia will install dependencies as per your Project.toml
### A package living in your local scratch
Pkg.add(path="/scratch/you/MYAPP.jl")
### A remote repository (public of private with your ssh key configured)
Pkg.add(path="https://github.com/someuser/MYAPP.jl.git")
### A registered package
Pkg.add("Distributions")
### Verify the installed code passes self tests.
Pkg.test("MYAPP")
 
# If needed
# Pkg.update()
println("Done")
}}
 
{{File
  |name=run_julia_ethereal.sh
  |lang="bash"
  |contents=
#!/bin/bash
#SBATCH --ntasks=100
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=1024M
#SBATCH --time=0-00:10
 
## Make sure your existing installation is not modified
mv ~/.julia ~/.juliaold
## Set package manager path
export JULIA_DEPOT_PATH="$SLURM_TMPDIR:$JULIA_DEPOT_PATH"
# Or
# export JULIA_DEPOT_PATH="$SLURM_TMPDIR"
 
## Setup dependencies (since your Julia install is now bare of any packages)
## This takes less time than e.g. updating existing installation in ~/.julia
julia julia_install.jl
 
## Run your code
julia mycode.jl
 
## Restore the old package directory
mv ~/.juliaold ~/.julia
 
## Restore the depot variable
export JULIA_DEPOT_PATH="" # Setting to empty will cause Julia to repopulate it, e.g. to ~/.julia
}}
 
 
This ensures any permanent registries and packages in $HOME/.julia are unaltered during execution of large compute tasks, while at the same time causing a drastic speedup.
For more information see the relevant section of the Julia [https://docs.julialang.org/en/v1/manual/faq/#How-do-I-manage-precompilation-caches-in-distributed-file-systems? documentation.]


= Available versions = <!--T:9-->  
= Available versions = <!--T:9-->  
Mboisson
Bureaucrats, cc_docs_admin, cc_staff, rsnt_translations
2,837

edits

Retrieved from "https://docs.alliancecan.ca/wiki/Special:MobileDiff/103161"

Navigation menu