Using Nix

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

Overview

Nix is a package manager system that allows users to manage their own persistent software environments. At the moment it is only available on SHARCNET systems (i.e., graham and legacy). If you would like this to change, help motivate an expansion by letting us known (it requires some coordination, but isn't too difficult to do).

• Supports one-off, per-project, and per-user usage of packages
• Packages can be built, installed, upgraded, downgraded, and removed as a user
• Operations either succeed or fail leaving everything intact (operations are atomic).
• Extremely each to add and share packages

Currently nix is building packages in a generic manner (e.g., without AVX2 or AVX512 vector instructions support), so module loaded software should be preferred for longer running simulations when it exists.

NOTE: The message failed to lock thread to CPU XX is a harmless warning that can be ignored.

Enabling and disabling the nix environment

The user’s current nix environment is enabled by loading the nix module. This creates some .nix* files and sets some environment variables.

[name@cluster:~]$ module load nix

It is disabled by unloading the nix module. This unsets the environment variables but leaves the .nix files alone.

[name@cluster:~]$ module unload nix

Completely resetting the nix environment

Most per-user operations can be undone with the --rollback option (i.e., nix-env --rollback or nix-channel --rollback). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

[name@cluster:~]$ module unload nix
[name@cluster:~]$ rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
[name@cluster:~]$ rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
[name@cluster:~]$ module load nix

Basic package usage

The nix search command can be used to locate available packages

[user@cluster:~]$ nix search git
...
* nixpkgs.git (git-minimal-2.19.3)
  Distributed version control system
...

Pro tips include

• you need to specify -u after upgrading your package set (this will take awhile)
• the search string is actually a regular expression and multiple ones are ANDed together

Often our usage of a package is either a one-off, a per-project, or an all the time situations. Nix supports all three of these cases.

One offs

If you just want to use a package once, the easiest was is to use the nix run command. This command will start a shell in which PATH has been extended to include the specified package

[user@cluster:~]$ nix run nixpkg.git
[user@cluster:~]$ git
[user@cluster:~]$ exit

Note that this does not protect the package from being garbage collected overnight (e.g., the package is only guaranteed to be around temporarily for your use until sometime in the wee-morning hours). Pro tips include

• you can specify more than one package in the same nix run command
• you can specify a command instead of a shell with -c <cmd> <args> ...

Per-project

If you want to use a program for a specific project, the easiest way is with the nix build command. This command will create a symbolic link (by default named result) from which you can access the programs bin directory to run it.

[user@cluster:~]$ nix build nixpkgs.git
[user@cluster:~]$ ./result/bin/git

Note that (currently) the package will only be protected from overnight garbage collection if you output the symlink into your home directory and do not rename or move it. Pro tips include

• you can specify the output symlink name with the -o <name> option
• add the bin directory to your PATH to not have to type it in every time

Per-user

Loading the nix module adds the per-user common ~/.nix-profile/bin directory to your PATH. You can add and remove packages from this directory with the nix-env command

[user@cluster:~]$ nix-env --install --attr nixpkgs.git
[user@cluster:~]$ nix-env --query
git-minimal-2.19.3

[user@cluster:~]$ nix-env --uninstall git-minimal
uninstalling 'git-minimal-2.19.3'
[user@cluster:~]$ nix-env --query

Each command actually creates a new version, so all prior versions remain and can be used

[user@cluster:~]$ nix-env --list-generations
   1   2020-07-29 13:10:03
   2   2020-07-29 13:11:52   (current)
[user@cluster:~]$ nix-env --switch-generation 1
[user@cluster:~]$ nix-env --query
git-minimal-2.19.3
[user@cluster:~]$ nix-env --switch-generation 2
[user@cluster:~]$ nix-env --query

Pro tips include

• nix-env --rollback moves up one generation
• nix-env --delete-generations <time> deletes environments older than <time> (e.g., 30d)
• see our nix-env page page for a much more in-depth discussion of using nix-env

Advanced package usage

Often we require a composition of packages. This can be as simple as having the binaries from multiple packages available in the same bin directory (e.g., make, gcc, and ld to build a simple C program) to as complex as having a python environment setup with all the desired modules installed (e.g., PYTHON_PATH set correctly, etc.).

All of these have a common format. You write a nix expression in a .nix file that composes together packages in a file and then you tell the above commands to use that with the -f <nix file> option. For example, say the file python.nix has an expression for a python environment in it, you can create a per-project bin directory with

[user@cluster:~]$ nix build -f python.nix -o python
[user@cluster:~]$ ./python/bin/python

The nix expression you put in the file generally

• does with import <nixpkgs> { } to bring the set of nixpkgs into scope
• calls an existing package composition functions with a list of space-separated components to include

The template for doing the second these follows below as it differs slightly across the various eco-systems.

A pro tip is

• there are many languages and framework supported but only a few described here, send us an email if you would like a missing supported one added here

Generic

Nixpkgs provides a buildEnv function that combine multiple packages into a single ones (by combining their bin, lib, etc. directories). The list of packages are the same as used before minus the leading nixpkgs as it was imported (e.g., git instead of nixpkgs.git).

with import <nixpkgs> {};
buildEnv {
  name = "my environment";
  paths = [
    ... list of packages ...
  ];
}

Python

Nixpkgs provides the following python related attributes

• python<major><minor> - a package providing the given python
• python<major><minor>.pkgs - the set of python packages using the given python
• python<major><minor>.withPackages - wraps python with PYTHON_PATH set to a given set of packages

We can use the former directly to use the programs provided by python packages

[user@cluster:~]$ nix run python36.pkgs.spambayes
[user@cluster:~]$ sb_filter.py --help
[user@cluster:~]$ exit

and the later in a .nix file to create a python wrapper to enable a given set of libraries (e.g., a python command we can run and access the given set of python packages from)

with import <nixpkgs> { };
python.withPackages (packages:
  with packages; [
    ... list of python packages ...
  ]
)

Some pro tips are

• the aliases python and python<major> given default python<major><minor> versions
• the aliases pythonPackages<major><minor> are short for python<major><minor>.pkgs (including default version variants)
• the function python<major><minor>.pkgs.buildPythonPackage can be used to build your own packages

R

Nixpkgs provides the following R related attributes

• R - a package providing R
• rstudio - a package providing RStudio
• rPackages - the set of R packages
• rWrapper - a wrapped R with R_LIBS set to a minimal set of packages
• rstudioWrapper - a wrapped RStudio with R_LIBS set to a minimal set of packages

We can use rPackages directly to examine the content of packages

[user@cluster:~]$ nix build rPackages.exams -o exams
[user@cluster:~]$ cat exams/library/exams/NEWS
[user@cluster:~]$ exit

and the latter two can be overridden in a .nix file to create R and RStudio wrappers to enable a given set of libraries (e.g., a R or rstudio command we can run and access the given set of R packages from)

with import <nixpkgs> { };
rWrapper.override {
  packages = with rPackages; [
    ... list of R packages ...
  ];
}

A pro tips is

• the function rPackages.buildRPackage can be used to build your own R packages

Haskell

Nixpkgs provides the following haskell related attributes

• haskell.compiler.ghc<major><minor><patch> - package providing the given ghc
• haskell.packages.ghc<major><minor><patch> - the set of haskell packages compiled by the given ghc
• haskell.packages.ghc<major><minor><patch>.withPackages - wraps ghc to enable the given packages
• haskell.packages.ghc<major><minor><patch>.withHoogle - wraps ghc to enable the given packages with hoogle and documentation indices

We can use the first directly to use programs provided by haskell packages

[user@cluster:~]$ nix run haskell.packages.ghc864.pandoc
[user@cluster:~]$ pandoc --help

and the last two in a .nix file create a ghc environment to enable a given set of package (e.g., a ghci we can run and access the given set of packages from)

with import <nixpkgs> { };
haskell.packages.ghc864.withPackages (packages:
  with packages; [
    ... list of Haskell packages ...
  ];
}

Some pro tips are

• the alias haskellPackages gives a default haskell.packages.ghc<major><minor><patch>
• the attributes in haskell.lib contains a variety of useful attributes for tweaking haskell packages (e.g., enabling profiling, etc.)
• the upstream maintainer has a useful youtube video on how to fix broken haskell packages

Emacs

Nixpkgs provides the following emacs related attributes (append a Ng suffix for older versions of nixpkgs, e.g., emacs25Ng and emacs25PackagesNg)

• emacs<major><minor> - a package providing the given emacs editor
• emacs<major><minor>Packages - the set of emacs packages for the given emacs editor
• emacs<major><minor>Packages.emacsWithPackages - wraps emacs to enable the given packages

We can use the second directly examine the content of packages

[user@cluster:~]$ nix build nixpkgs.emacs25Packages.magit -o magit
[user@cluster:~]$ cat magit/share/emacs/site-lisp/elpa/magit*/AUTHORS.md
[user@cluster:~]$ exit

and the last one in a .nix file create an emacs with the given set of packages enabled

with import <nixpkgs> { };
emacs25Packages.emacsWithPackages (packages:
  with packages; [
    ... list of emacs packages ...
  ];
}

Some pro tips are

• the aliases emacs and emacsPackages give a default emacs<major><minor> and emacsPackages<major><minor> version
• the alias emacs<major><minor>WithPackages are short for emacs<major><minor>Packages.emacsWithPackages (including default version variants)