cc_staff
40
edits
Using Nix: Difference between revisions
Jump to navigation
Jump to search
no edit summary
(Morning after cleanups) |
No edit summary |
||
| Line 3: | Line 3: | ||
= Overview = | = Overview = | ||
[https://nixos.org/nix/ Nix] is a | [https://nixos.org/nix/ Nix] is a software building and composition system that allows users to manage their own persistent software environments. At the moment it is only available on SHARCNET systems (i.e., graham and legacy). If you would like this to change, let us know (it requires some coordination, but isn’t too difficult to do). | ||
* Supports one-off, per-project, and per-user usage of | * Supports one-off, per-project, and per-user usage of compositions | ||
* | * Compositions can be built, installed, upgraded, downgraded, and removed as a user | ||
* Operations either succeed or fail leaving everything intact (operations are atomic). | * Operations either succeed or fail leaving everything intact (operations are atomic). | ||
* Extremely | * Extremely easy to add and share compositions | ||
Currently nix is building | Currently nix is building software in a generic manner (e.g., without AVX2 or AVX512 vector instructions support), so module loaded software should be preferred for longer running simulations when it exists. | ||
'''NOTE:''' The message <code>failed to lock thread to CPU XX</code> is a harmless warning that can be ignored. | '''NOTE:''' The message <code>failed to lock thread to CPU XX</code> is a harmless warning that can be ignored. | ||
| Line 19: | Line 19: | ||
<source lang="bash">[name@cluster:~]$ module load nix</source> | <source lang="bash">[name@cluster:~]$ module load nix</source> | ||
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix'' files alone. | It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone. | ||
<source lang="bash">[name@cluster:~]$ module unload nix</source> | <source lang="bash">[name@cluster:~]$ module unload nix</source> | ||
| Line 30: | Line 30: | ||
[name@cluster:~]$ rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER | [name@cluster:~]$ rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER | ||
[name@cluster:~]$ module load nix</source> | [name@cluster:~]$ module load nix</source> | ||
= | = Existing compositions = | ||
The <code>nix search</code> command can be used to locate available | The <code>nix search</code> command can be used to locate already available compositions | ||
<source lang="bash">[user@cluster:~]$ nix search git | <source lang="bash">[user@cluster:~]$ nix search git | ||
| Line 41: | Line 41: | ||
Pro tips include | Pro tips include | ||
* you need to specify <code>-u</code> after upgrading your | * you need to specify <code>-u</code> after upgrading your channel (this will take awhile) | ||
* the search string is actually a regular expression and multiple ones are ANDed together | * the search string is actually a regular expression and multiple ones are ANDed together | ||
Often our usage of a | Often our usage of a composition is either a one-off, a per-project, or an all the time situations. Nix supports all three of these cases. | ||
== One offs == | == One offs == | ||
If you just want to use a | If you just want to use a composition once, the easiest was is to use the <code>nix run</code> command. This command will start a shell in which <code>PATH</code> has been extended to include the specified composition | ||
<source lang="bash">[user@cluster:~]$ nix run nixpkg.git | <source lang="bash">[user@cluster:~]$ nix run nixpkg.git | ||
[user@cluster:~]$ git | [user@cluster:~]$ git | ||
[user@cluster:~]$ exit</source> | [user@cluster:~]$ exit</source> | ||
Note that this does not protect the | Note that this does not protect the composition from being garbage collected overnight (e.g., the composition is only guaranteed to be around temporarily for your use until sometime in the wee-morning hours). Pro tips include | ||
* you can specify more than one | * you can specify more than one composition in the same <code>nix run</code> command | ||
* you can specify a command instead of a shell with <code>-c <cmd> <args> ...</code> | * you can specify a command instead of a shell with <code>-c <cmd> <args> ...</code> | ||
== Per-project == | == Per-project == | ||
If you want to use a program for a specific project, the easiest way is with the <code>nix build</code> command. This command will create a symbolic link (by default named <code>result</code>) from which you can access the programs | If you want to use a program for a specific project, the easiest way is with the <code>nix build</code> command. This command will create a symbolic link (by default named <code>result</code>) from which you can access the programs ''bin'' directory to run it. | ||
<source lang="bash">[user@cluster:~]$ nix build nixpkgs.git | <source lang="bash">[user@cluster:~]$ nix build nixpkgs.git | ||
[user@cluster:~]$ ./result/bin/git</source> | [user@cluster:~]$ ./result/bin/git</source> | ||
Note that (currently) the | Note that (currently) the composition will only be protected from overnight garbage collection if you output the symlink into your ''home'' directory and do not rename or move it. Pro tips include | ||
* you can specify the output symlink name with the <code>-o <name></code> option | * you can specify the output symlink name with the <code>-o <name></code> option | ||
* add the | * add the ''bin'' directory to your <code>PATH</code> to not have to type it in every time | ||
== Per-user == | == Per-user == | ||
Loading the <code>nix</code> module adds the per-user common | Loading the <code>nix</code> module adds the per-user common ''~/.nix-profile/bin'' directory to your <code>PATH</code>. You can add and remove compositions from this directory with the <code>nix-env</code> command | ||
<source lang="bash">[user@cluster:~]$ nix-env --install --attr nixpkgs.git | <source lang="bash">[user@cluster:~]$ nix-env --install --attr nixpkgs.git | ||
| Line 93: | Line 93: | ||
* <code>nix-env --rollback</code> moves up one generation | * <code>nix-env --rollback</code> moves up one generation | ||
* <code>nix-env --delete-generations <time></code> deletes environments older than <code><time></code> (e.g., <code>30d</code>) | * <code>nix-env --delete-generations <time></code> deletes environments older than <code><time></code> (e.g., <code>30d</code>) | ||
* see our [[ | * see our [[nix-env.md|nix-env page]] for a much more in-depth discussion of using <code>nix-env</code> | ||
= | = Creating compositions = | ||
Often we require | Often we require our own unique composition. A basic example would be to bundle all the binaries from multiple existing compositions in a common ''bin'' directory (e.g., <code>make</code>, <code>gcc</code>, and <code>ld</code> to build a simple C program). A more complex example would be to bundle python with a set of python libraries by wrapping the python executables with shell scripts to set <code>PYTHON_PATH</code> for the python libraries before running the real python binaries. | ||
All of these have a common format. You write a nix expression in a <code>.nix</code> file that composes together | All of these have a common format. You write a nix expression in a <code>.nix</code> file that composes together existing compositions and then you tell the above commands to use that with the <code>-f <nix file></code> option. For example, say the file <code>python.nix</code> has an expression for a python environment in it, you can create a per-project ''bin'' directory with | ||
<source lang="bash">[user@cluster:~]$ nix build -f python.nix -o python | <source lang="bash">[user@cluster:~]$ nix build -f python.nix -o python | ||
| Line 105: | Line 105: | ||
The nix expression you put in the file generally | The nix expression you put in the file generally | ||
* does <code>with import <nixpkgs> { }</code> to bring the set of nixpkgs into scope | * does <code>with import <nixpkgs> {}</code> to bring the set of nixpkgs into scope | ||
* calls an existing | * calls an existing composition functions with a list of space-separated components to include | ||
The template for doing the second these follows below as it differs slightly across the various eco-systems. | The template for doing the second these follows below as it differs slightly across the various eco-systems. | ||
| Line 116: | Line 116: | ||
== Generic == | == Generic == | ||
Nixpkgs provides a <code>buildEnv</code> function that | Nixpkgs provides a <code>buildEnv</code> function that does a basic composition of compositions (by combining their ''bin'', ''lib'', etc. directories). The list of packages are the same as used before minus the leading <code>nixpkgs</code> prefix as it was imported (e.g., <code>git</code> instead of <code>nixpkgs.git</code>). | ||
<source lang="nix">with import <nixpkgs> {}; | <source lang="nix">with import <nixpkgs> {}; | ||
| Line 122: | Line 122: | ||
name = "my environment"; | name = "my environment"; | ||
paths = [ | paths = [ | ||
... list of | ... list of compositions ... | ||
]; | ]; | ||
}</source> | }</source> | ||
| Line 129: | Line 129: | ||
Nixpkgs provides the following python related attributes | Nixpkgs provides the following python related attributes | ||
* <code>python<major><minor></code> - a | * <code>python<major><minor></code> - a composition providing the given python | ||
* <code>python<major><minor>.pkgs</code> - the set of python | * <code>python<major><minor>.pkgs</code> - the set of python compositions using the given python | ||
* <code>python<major><minor>.withPackages</code> - wraps python with <code>PYTHON_PATH</code> set to a given set of packages | * <code>python<major><minor>.withPackages</code> - wraps python with <code>PYTHON_PATH</code> set to a given set of python packages | ||
We can use the former directly to use the programs provided by python | We can use the former directly to use the programs provided by python compositions | ||
<source lang="bash">[user@cluster:~]$ nix run python36.pkgs.spambayes | <source lang="bash">[user@cluster:~]$ nix run python36.pkgs.spambayes | ||
[user@cluster:~]$ sb_filter.py --help | [user@cluster:~]$ sb_filter.py --help | ||
[user@cluster:~]$ exit</source> | [user@cluster:~]$ exit</source> | ||
and the later in a <code>.nix</code> file to create a python | and the later in a <code>.nix</code> file to create a python composition that enables a given set of libraries (e.g., a <code>python</code> command we can run and access the given set of python packages from) | ||
<source lang="nix">with import <nixpkgs> { }; | <source lang="nix">with import <nixpkgs> { }; | ||
| Line 150: | Line 150: | ||
* the aliases <code>python</code> and <code>python<major></code> given default <code>python<major><minor></code> versions | * the aliases <code>python</code> and <code>python<major></code> given default <code>python<major><minor></code> versions | ||
* the aliases <code>pythonPackages<major><minor></code> are short for <code>python<major><minor>.pkgs</code> (including default version variants) | * the aliases <code>pythonPackages<major><minor></code> are short for <code>python<major><minor>.pkgs</code> (including default version variants) | ||
* the function <code>python<major><minor>.pkgs.buildPythonPackage</code> can be used to build your own packages | * the function <code>python<major><minor>.pkgs.buildPythonPackage</code> can be used to build your own python packages | ||
== R == | == R == | ||
| Line 156: | Line 156: | ||
Nixpkgs provides the following R related attributes | Nixpkgs provides the following R related attributes | ||
* <code>R</code> - a | * <code>R</code> - a composition providing R | ||
* <code>rstudio</code> - a | * <code>rstudio</code> - a composition providing RStudio | ||
* <code>rPackages</code> - the set of R packages | * <code>rPackages</code> - the set of R packages | ||
* <code>rWrapper</code> - a | * <code>rWrapper</code> - a composition that wraps R with <code>R_LIBS</code> set to a minimal set of R packages | ||
* <code>rstudioWrapper</code> - a wrapped RStudio with <code>R_LIBS</code> set to a minimal set of packages | * <code>rstudioWrapper</code> - a composition that wrapped RStudio with <code>R_LIBS</code> set to a minimal set of R packages | ||
We can use <code>rPackages</code> directly to examine the content of packages | We can use <code>rPackages</code> directly to examine the content of R packages | ||
<source lang="bash">[user@cluster:~]$ nix build rPackages.exams -o exams | <source lang="bash">[user@cluster:~]$ nix build rPackages.exams -o exams | ||
[user@cluster:~]$ cat exams/library/exams/NEWS | [user@cluster:~]$ cat exams/library/exams/NEWS | ||
[user@cluster:~]$ exit</source> | [user@cluster:~]$ exit</source> | ||
and the latter two can be overridden in a <code>.nix</code> file to create R and RStudio wrappers to | and the latter two can be overridden in a <code>.nix</code> file to create R and RStudio wrappers to create a composition enabling a given set of R libraries (e.g., a <code>R</code> or <code>rstudio</code> command we can run and access the given set of R packages from) | ||
<source lang="nix">with import <nixpkgs> { }; | <source lang="nix">with import <nixpkgs> { }; | ||
| Line 183: | Line 183: | ||
Nixpkgs provides the following haskell related attributes | Nixpkgs provides the following haskell related attributes | ||
* <code>haskell.compiler.ghc<major><minor><patch></code> - | * <code>haskell.compiler.ghc<major><minor><patch></code> - composition providing the given ghc | ||
* <code>haskell.packages.ghc<major><minor><patch></code> - the set of haskell packages compiled by the given ghc | * <code>haskell.packages.ghc<major><minor><patch></code> - the set of haskell packages compiled by the given ghc | ||
* <code>haskell.packages.ghc<major><minor><patch>.withPackages</code> - | * <code>haskell.packages.ghc<major><minor><patch>.withPackages</code> - composition wrapping ghc to enable the given packages | ||
* <code>haskell.packages.ghc<major><minor><patch>.withHoogle</code> - | * <code>haskell.packages.ghc<major><minor><patch>.withHoogle</code> - composition wrapping ghc to enable the given packages with hoogle and documentation indices | ||
We can use the first directly to use programs provided by haskell packages | We can use the first directly to use programs provided by haskell packages | ||
| Line 192: | Line 192: | ||
<source lang="bash">[user@cluster:~]$ nix run haskell.packages.ghc864.pandoc | <source lang="bash">[user@cluster:~]$ nix run haskell.packages.ghc864.pandoc | ||
[user@cluster:~]$ pandoc --help</source> | [user@cluster:~]$ pandoc --help</source> | ||
and the last two in a <code>.nix</code> file create a ghc environment to enable a given set of package (e.g., a <code>ghci</code> we can run and access the given set of packages from) | and the last two in a <code>.nix</code> file create a ghc environment to enable a given set of haskell package (e.g., a <code>ghci</code> we can run and access the given set of packages from) | ||
<pre>with import <nixpkgs> { }; | <pre>with import <nixpkgs> { }; | ||
| Line 210: | Line 210: | ||
Nixpkgs provides the following emacs related attributes (append a <code>Ng</code> suffix for older versions of nixpkgs, e.g., <code>emacs25Ng</code> and <code>emacs25PackagesNg</code>) | Nixpkgs provides the following emacs related attributes (append a <code>Ng</code> suffix for older versions of nixpkgs, e.g., <code>emacs25Ng</code> and <code>emacs25PackagesNg</code>) | ||
* <code>emacs<major><minor></code> - a | * <code>emacs<major><minor></code> - a composition providing the given emacs editor | ||
* <code>emacs<major><minor>Packages</code> - the set of emacs packages for the given emacs editor | * <code>emacs<major><minor>Packages</code> - the set of emacs packages for the given emacs editor | ||
* <code>emacs<major><minor>Packages.emacsWithPackages</code> - | * <code>emacs<major><minor>Packages.emacsWithPackages</code> - composition wrapping emacs to enable the given packages | ||
We can use the second directly examine the content of packages | We can use the second directly examine the content of packages | ||
| Line 219: | Line 219: | ||
[user@cluster:~]$ cat magit/share/emacs/site-lisp/elpa/magit*/AUTHORS.md | [user@cluster:~]$ cat magit/share/emacs/site-lisp/elpa/magit*/AUTHORS.md | ||
[user@cluster:~]$ exit</source> | [user@cluster:~]$ exit</source> | ||
and the last one in a <code>.nix</code> file create | and the last one in a <code>.nix</code> file create a composition giving emacs with the given set of packages enabled | ||
<pre>with import <nixpkgs> { }; | <pre>with import <nixpkgs> { }; | ||