8
edits
Julia: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
No edit summary |
||
| Line 7: | Line 7: | ||
<!--T:2--> | <!--T:2--> | ||
[https://julialang.org Julia] is a programming language that was designed from the beginning for performance, ease of use and portability. It | [https://julialang.org Julia] is a programming language that was designed from the beginning for performance, ease of use and portability. It is available as a [[Utiliser_des_modules/en | module]] on Compute Canada clusters. | ||
= Compiling packages = <!--T:3--> | = Compiling packages = <!--T:3--> | ||
| Line 32: | Line 32: | ||
<!--T:8--> | <!--T:8--> | ||
In the example above, installing just the JLD package creates a <code>~/.julia</code> tree with 18673 files and directories and using 236M of space, almost 5% of a standard user's quota for <code>/home</code>. It's worth remembering that installing a lot of packages will consume a lot of space. | In the example above, installing just the JLD package creates a <code>~/.julia</code> tree with 18673 files and directories and using 236M of space, almost 5% of a standard user's quota for <code>/home</code>. It's worth remembering that installing a lot of packages will consume a lot of space. | ||
In addition, because all Julia's user files are saved in $HOME/.julia, significant slowdowns can be experienced when working with non-trivial dependencies. When one uses Julia's distributed programming mode, this overhead is increased due to k Julia processes accessing $HOME/.julia/compiled. On interactive (salloc) or sbatch nodes, one can opt for a workaround by: | |||
{{Command | |||
|export JULIA_DEPOT_PATH="$SLURM_TMPDIR:$JULIA_DEPOT_PATH" | |||
}} | |||
With $SLURM_TMPDIR residing on a very fast local disk, installation, updates, precompilation, and reads to registries are much faster (up to x100). To make Julia only use this temporary path, one can use the following workaround | |||
{{File | |||
|name=julia_install.jl | |||
|lang="julia" | |||
|contents= | |||
import Pkg | |||
## Create local env in current dir (before you add packages) | |||
## If current dir has an existing (local) env | |||
# Pkg.activate(".") | |||
## Else | |||
Pkg.instantiate() | |||
## Add your application(s), julia will install dependencies as per your Project.toml | |||
### A package living in your local scratch | |||
Pkg.add(path="/scratch/you/MYAPP.jl") | |||
### A remote repository (public of private with your ssh key configured) | |||
Pkg.add(path="https://github.com/someuser/MYAPP.jl.git") | |||
### A registered package | |||
Pkg.add("Distributions") | |||
### Verify the installed code passes self tests. | |||
Pkg.test("MYAPP") | |||
# If needed | |||
# Pkg.update() | |||
println("Done") | |||
}} | |||
{{File | |||
|name=run_julia_ethereal.sh | |||
|lang="bash" | |||
|contents= | |||
#!/bin/bash | |||
#SBATCH --ntasks=100 | |||
#SBATCH --cpus-per-task=4 | |||
#SBATCH --mem-per-cpu=1024M | |||
#SBATCH --time=0-00:10 | |||
## Make sure your existing installation is not modified | |||
mv ~/.julia ~/.juliaold | |||
## Set package manager path | |||
export JULIA_DEPOT_PATH="$SLURM_TMPDIR:$JULIA_DEPOT_PATH" | |||
# Or | |||
# export JULIA_DEPOT_PATH="$SLURM_TMPDIR" | |||
## Setup dependencies (since your Julia install is now bare of any packages) | |||
## This takes less time than e.g. updating existing installation in ~/.julia | |||
julia julia_install.jl | |||
## Run your code | |||
julia mycode.jl | |||
## Restore the old package directory | |||
mv ~/.juliaold ~/.julia | |||
## Restore the depot variable | |||
export JULIA_DEPOT_PATH="" # Setting to empty will cause Julia to repopulate it, e.g. to ~/.julia | |||
}} | |||
This ensures any permanent registries and packages in $HOME/.julia are unaltered during execution of large compute tasks, while at the same time causing a drastic speedup. | |||
For more information see the relevant section of the Julia [https://docs.julialang.org/en/v1/manual/faq/#How-do-I-manage-precompilation-caches-in-distributed-file-systems? documentation.] | |||
= Available versions = <!--T:9--> | = Available versions = <!--T:9--> | ||