Home » Archive by category "nixpills"

Nix pill 19: fundamentals of stdenv

Welcome to the 19th Nix pill. In the previous 18th pill we did dive into the algorithm used by Nix to compute the store paths, and also introduced fixed-output store paths.This time we will instead look into nixpkgs, in particular one of its core deriv...

Nix pill 18: nix store paths

Welcome to the 18th Nix pill. In the previous 17th pill we have scratched the surface of the nixpkgs repository structure. It is a set of packages, and it's possible to override such packages so that all other packages will use the overrides.

Before reading existing derivations, I'd like to talk about store paths and how they are computed. In particular we are interested in fixed store paths that depend on an integrity hash (e.g. a sha256), which is usually applied to source tarballs.

The way store paths are computed is a little contrived, mostly due to historical reasons. Our reference will be the Nix source code.

Source paths


Let's start simple. You know nix allows relative paths to be used, such that the file or directory is stored in the nix store, that is ./myfile gets stored into /nix/store/....... We want to understand how is the store path generated for such a file:
$ echo mycontent > myfile
I remind you, the simplest derivation you can write has a name, a builder and the system:
$ nix-repl
nix-repl> derivation { system = "x86_64-linux"; builder = ./myfile; name = "foo"; }
«derivation /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv»
Now inspect the .drv to see where is ./myfile being stored:
$ pp-aterm -i /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv
Derive(
[("out", "/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo", "", "")]
, []
, ["/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile"]
, "x86_64-linux"
...
Great, how did nix decide to use xv2iccirbrvklck36f1g7vldn5v58vck ? Keep looking at the nix comments.

Note: doing nix-store --add myfile will store the file in the same store path.

Step 1, compute the hash of the file


The comments tell us to first compute the sha256 of the NAR serialization of the file. Can be done in two ways:
$ nix-hash --type sha256 myfile
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3
Or:
$ nix-store --dump myfile|sha256sum
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3 -
In general, Nix understands two contents: flat for regular files, or recursive for NAR serializations which can be anything.

Step 2, build the string description


Then nix uses a special string which includes the hash, the path type and the file name. We store this in another file:
$ echo -n "source:sha256:2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3:/nix/store:myfile" > myfile.str

Step 3, compute the final hash


Finally the comments tell us to compute the base-32 representation of the first 160 bits (truncation) of a sha256 of the above string:
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
xv2iccirbrvklck36f1g7vldn5v58vck

Output paths


Output paths are usually generated for derivations. We use the above example because it's simple. Even if we didn't build the derivation, nix knows the out path hs0yi5n5nw6micqhy8l1igkbhqdkzqa1. This is because the out path only depends on inputs.

It's computed in a similar way to source paths, except that the .drv is hashed and the type of derivation is output:out. In case of multiple outputs, we may have different output:<id>.

At the time nix computes the out path, the .drv contains an empty string for each out path. So what we do is getting our .drv and replacing the out path with an empty string:
$ cp -f /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv myout.drv
$ sed -i 's,/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo,,g' myout.drv
The myout.drv is the .drv state in which nix is when computing the out path for our derivation:
$ sha256sum myout.drv
1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5 myout.drv
$ echo -n "output:out:sha256:1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5:/nix/store:foo" > myout.str
$ nix-hash --type sha256 --truncate --base32 --flat myout.str
hs0yi5n5nw6micqhy8l1igkbhqdkzqa1
Then nix puts that out path in the .drv, and that's it.

In case the .drv has input derivations, that is it references other .drv, then such .drv paths are replaced by this same algorithm which returns an hash.

In other words, you get a final .drv where every other .drv path is replaced by its hash.

Fixed-output paths


Finally, the other most used kind of path is when we know beforehand an integrity hash of a file. This is usual for tarballs.

A derivation can take three special attributes: outputHashMode, outputHash and outputHashAlgo which are well documented in the nix manual.

The builder must create the out path and make sure its hash is the same as the one declared with outputHash.

Let's say our builder should create a file whose contents is mycontent:
$ echo mycontent > myfile
$ sha256sum myfile
f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb myfile
nix-repl> derivation { name = "bar"; system = "x86_64-linux"; builder = "none"; outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb"; }
«derivation /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv»
Inspect the .drv and see that it also stored the fact that it's a fixed-output derivation with sha256 algorithm, compared to the previous examples:
$ pp-aterm -i /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv
Derive(
[("out", "/nix/store/a00d5f71k0vp5a6klkls0mvr1f7sx6ch-bar", "sha256", "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb")]
...
It doesn't matter which input derivations are being used, the final out path must only depend on the declared hash.
What nix does is to create an intermediate string representation of the fixed-output content:
$ echo -n "fixed:out:sha256:f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb:" > mycontent.str
$ sha256sum mycontent.str
423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639 myfile.str
Then proceed as it was a normal derivation output path:
$ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639:/nix/store:bar" > myfile.str
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
a00d5f71k0vp5a6klkls0mvr1f7sx6ch
Hence, the store path only depends on the declared fixed-output hash.

Conclusion


There are other types of store paths, but you get the idea. Nix first hashes the contents, then creates a string description, and the final store path is the hash of this string.

Also we've introduced some fundamentals, in particular the fact that Nix knows beforehand the out path of a derivation since it only depends on the inputs. We've also introduced fixed-output derivations which are especially used by the nixpkgs repository for downloading and verifying source tarballs.

Next pill


...we will introduce stdenv. In the previous pills we rolled our own mkDerivation convenience function for wrapping the builtin derivation, but the nixpkgs repository also has its own convenience functions for dealing with autotools projects and other build systems.

Pill 19 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Nix pill 17: nixpkgs, overriding packages

Welcome to the 17th Nix pill. In the previous 16th pill we have started to dive into the nixpkgs repository. Nixpkgs is a function, and we've looked at some parameters like system and config.Today we'll talk about a special attribute: config.packageOve...

Nix pill 16: nixpkgs, the parameters

Welcome to the 16th Nix pill. In the previous 15th pill we've realized how nix finds expressions with the angular brackets syntax, so that we finally know where is <nixpkgs> located on our system.We can start diving into the nixpkg...

Nix pill 15: nix search paths

Welcome to the 15th Nix pill. In the previous 14th pill we have introduced the "override" pattern, useful for writing variants of derivations by passing different inputs.Assuming you followed the previous posts, I hope you are now ready to un...

Nix pill 14: the override design pattern

Welcome to the 14th Nix pill. In the previous 13th pill we have introduced the callPackage pattern, used to simplify the composition of software in a repository.

The next design pattern is less necessary but useful in many cases and it's a good exercise to learn more about Nix.

About composability


Functional languages are known for being able to compose functions. In particular, you gain a lot from functions that are able to manipulate the original value into a new value having the same structure. So that in the end we're able to call multiple functions to have the desired modifications.

In Nix we mostly talk about functions that accept inputs in order to return derivations. In our world we want nice utility functions that are able to manipulate those structures. These utilities add some useful properties to the original value, and we must be able to apply more utilities on top of it.

For example let's say we have an initial derivation drv and we want it to be a drv with debugging information and also to apply some custom patches:
debugVersion (applyPatches [ ./patch1.patch ./patch2.patch ] drv)
The final result will be still the original derivation plus some changes. That's both interesting and very different from other packaging approaches, which is a consequence of using a functional language to describe packages.

Designing such utilities is not trivial in a functional language that is not statically typed, because understanding what can or cannot be composed is difficult. But we try to do the best.

The override pattern


In the pill 12 we introduced the inputs design pattern. We do not return a derivation picking dependencies directly from the repository, rather we declare the inputs and let the callers pass the necessary arguments.

In our repository we have a set of attributes that import the expressions of the packages and pass these arguments, getting back a derivation. Let's take for example the graphviz attribute:
graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; };
If we wanted to produce a derivation of graphviz with a customized gd version, we would have to repeat most of the above plus specifying an alternative gd:
mygraphviz = import ./graphviz.nix {
inherit mkDerivation fontconfig libjpeg bzip2;
gd = customgd;
};
That's hard to maintain. Using callPackage it would be easier:
mygraphviz = callPackage ./graphviz.nix { gd = customgd; };
But we may still be diverging from the original graphviz in the repository.

We would like to avoid specifying the nix expression again, instead reuse the original graphviz attribute in the repository and add our overrides like this:
mygraphviz = graphviz.override { gd = customgd; };
The difference is obvious, as well as the advantages of this approach.

Note: that .override is not a "method" in the OO sense as you may think. Nix is a functional language. That .override is simply an attribute of a set.

The override implementation


I remind you, the graphviz attribute in the repository is the derivation returned by the function imported from graphviz.nix. We would like to add a further attribute named "override" to the returned set.

Let's start simple by first creating a function "makeOverridable" that takes a function and a set of original arguments to be passed to the function.

Contract: the wrapped function must return a set.

Let's write a lib.nix:
{
makeOverridable = f: origArgs:
let
origRes = f origArgs;
in
origRes // { override = newArgs: f (origArgs // newArgs); };
}
So makeOverridable takes a function and a set of original arguments. It returns the original returned set, plus a new override attribute.
This override attribute is a function taking a set of new arguments, and returns the result of the original function called with the original arguments unified with the new arguments. What a mess.

Let's try it with nix-repl:
$ nix-repl
nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> f { a = 3; b = 5; }
{ result = 8; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res
{ override = «lambda»; result = 8; }
nix-repl> res.override { a = 10; }
{ result = 15; }
Note that the function f does not return the plain sum but a set, because of the contract. You didn't forget already, did you? :-)
The variable res is the result of the function call without any override. It's easy to see in the definition of makeOverridable. In addition you can see the new override attribute being a function.

Calling that .override with a set will invoke the original function with the overrides, as expected.
But: we can't override again! Because the returned set with result 15 does not have an override attribute!
That's bad, it breaks further compositions.

The solution is simple, the .override function should make the result overridable again:
rec {
makeOverridable = f: origArgs:
let
origRes = f origArgs;
in
origRes // { override = newArgs: makeOverridable f (origArgs // newArgs); };
}
Please note the rec keyword. It's necessary so that we can refer to makeOverridable from makeOverridable itself.

Now let's try overriding twice:
nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res2 = res.override { a = 10; }
nix-repl> res2
{ override = «lambda»; result = 15; }
nix-repl> res2.override { b = 20; }
{ override = «lambda»; result = 30; }

Success! The result is 30, as expected because a is overridden to 10 in the first override, and b to 20.

Now it would be nice if callPackage made our derivations overridable. That was the goal of this pill after all. This is an exercise for the reader.

Conclusion


The "override" pattern simplifies the way we customize packages starting from an existing set of packages. This opens a world of possibilities about using a central repository like nixpkgs, and defining overrides on our local machine without even modifying the original package.

Dream of a custom isolated nix-shell environment for testing graphviz with a custom gd:
debugVersion (graphviz.override { gd = customgd; })
Once a new version of the overridden package comes out in the repository, the customized package will make use of it automatically.

The key in Nix is to find powerful yet simple abstractions in order to let the user customize his environment with highest consistency and lowest maintenance time, by using predefined composable components.

Next pill


...we will talk about Nix search paths. By search path I mean a place in the file system where Nix looks for expressions. You may have wondered, where does that holy <nixpkgs> come from?

Pill 15 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Nix pill 13: the callPackage design pattern

Welcome to the 13th Nix pill. In the previous 12th pill we have introduced the first basic design pattern for organizing a repository of software. In addition we packaged graphviz to have at least another package for our little repository.The...

Nix pill 12: the inputs design pattern

Welcome to the 12th Nix pill. In the previous 11th pill we stopped packaging and cleaned up the system with the garbage collector.

We restart our packaging, but we will improve a different aspect. We only packaged an hello world program so far, what if we want to create a repository of multiple packages?

Repositories in Nix


Nix is a tool for build and deployment, it does not enforce any particular repository format. A repository of packages is the main usage for Nix, but not the only possibility. See it more like a consequence due to the need of organizing packages.
Nix is a language, and it is powerful enough to let you choose the format of your own repository. In this sense, it is not declarative, but functional.
There is no preset directory structure or preset packaging policy. It's all about you and Nix.

The nixpkgs repository has a certain structure, which evolved and evolves with the time. Like other languages, Nix has its own history and therefore I'd like to say that it also has its own design patterns. Especially when packaging, you often do the same task again and again except for different software. It's inevitable to identify patterns during this process. Some of these patterns get reused if the community thinks it's a good way to package the software.

Some of the patterns I'm going to show do not apply only to Nix, but to other systems of course.

The single repository pattern


Before introducing the "inputs" pattern, we can start talking about another pattern first which I'd like to call "single repository" pattern.
Systems like Debian scatter packages in several small repositories. Personally, this makes it hard to track interdependent changes and to contribute to new packages.
Systems like Gentoo instead, put package descriptions all in a single repository.

The nix reference for packages is nixpkgs, a single repository of all descriptions of all packages. I find this approach very natural and attractive for new contributions.

From now on, we will adopt this technique. The natural implementation in Nix is to create a top-level Nix expression, and one expression for each package. The top-level expression imports and combines all expressions in a giant attribute set with name -> package pairs.

But isn't that heavy? It isn't, because Nix is a lazy language, it evaluates only what's needed! And that's why nixpkgs is able to maintain such a big software repository in a giant attribute set.

Packaging graphviz


We have packaged GNU hello world, I guess you would like to package something else for creating at least a repository of two projects :-) . I chose graphviz, which uses the standard autotools build system, requires no patching and dependencies are optional.

Download graphviz from here. The graphviz.nix expression is straightforward:
let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
}
Build with nix-build graphviz.nix and you will get runnable binaries under result/bin. Notice how we did reuse the same autotools.nix of hello.nix. Let's create a simple png:
$ echo 'graph test { a -- b }'|result/bin/dot -Tpng -o test.png
Format: "png" not recognized. Use one of: canon cmap [...]
Oh of course... graphviz can't know about png. It built only the output formats it supports natively, without using any extra library.

I remind you, in autotools.nix there's a buildInputs variablewhich gets concatenated to baseInputs.  That would be the perfect place to add a build dependency. We created that variable exactly for this reason to be overridable from package expressions.

This 2.38 version of graphviz has several plugins to output png. For simplicity, we will use libgd.

Digression about gcc and ld wrappers


The gd, jpeg, fontconfig and bzip2 libraries (dependencies of gd) don't use pkg-config to specify which flags to pass to the compiler. Since there's no global location for libraries, we need to tell gcc and ld where to find includes and libs.

The nixpkgs provides gcc and binutils, and we are using them for our packaging. Not only, it also provides wrappers for them which allow passing extra arguments to gcc and ld, bypassing the project build systems:
  • NIX_CFLAGS_COMPILE: extra flags to gcc at compile time
  • NIX_LDFLAGS: extra flags to ld
What can we do about it? We can employ the same trick we did for PATH: automatically filling the variables from buildInputs. This is the relevant snippet of setup.sh:
for p in $baseInputs $buildInputs; do
if [ -d $p/bin ]; then
export PATH="$p/bin${PATH:+:}$PATH"
fi
if [ -d $p/include ]; then
export NIX_CFLAGS_COMPILE="-I $p/include${NIX_CFLAGS_COMPILE:+ }$NIX_CFLAGS_COMPILE"
fi
if [ -d $p/lib ]; then
export NIX_LDFLAGS="-rpath $p/lib -L $p/lib${NIX_LDFLAGS:+ }$NIX_LDFLAGS"
fi
done
Now by adding derivations to buildInputs, will add the lib, include and bin paths automatically in setup.sh.

The -rpath flag in ld is needed because at runtime, the executable must use exactly that version of the library.
If unneeded paths are specified, the fixup phase will shrink the rpath for us!

Completing graphviz with gd


Finish the expression for graphviz with gd support (note the use of the with expression in buildInputs to avoid repeating pkgs):
let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
buildInputs = with pkgs; [ gd fontconfig libjpeg bzip2 ];
}
Now you can create the png! Ignore any error from fontconfig, especially if you are in a chroot.

The repository expression


Now that we have two packages, what's a good way to put them together in a single repository? We do something like nixpkgs does. With nixpkgs, we import it and then we peek derivations by accessing the giant attribute set.
For us nixers, this a good technique, because it abstracts from the file names. We don't refer to a package by REPO/some/sub/dir/package.nix but by importedRepo.package (or pkgs.package in our examples).

Create a default.nix in the current directory:
{
hello = import ./hello.nix;
graphviz = import ./graphviz.nix;
}
Ready to use! Try it with nix-repl:
$ nix-repl
nix-repl> :l default.nix
Added 2 variables.
nix-repl> hello
«derivation /nix/store/dkib02g54fpdqgpskswgp6m7bd7mgx89-hello.drv»
nix-repl> graphviz
«derivation /nix/store/zqv520v9mk13is0w980c91z7q1vkhhil-graphviz.drv»
With nix-build:
$ nix-build default.nix -A hello
[...]
$ result/bin/hello
Hello, world!
The -A argument is used to access an attribute of the set from the given .nix expression.
Important: why did we choose the default.nix? Because when a directory (by default the current directory) has a default.nix, that default.nix will be used (see import here). In fact you can run nix-build -A hello without specifying default.nix. 
For pythoners, it is similar to __init__.py.

With nix-env, to install the package in your user environment:
$ nix-env -f . -iA graphviz
[...]
$ dot -V
The -f option is used to specify the expression to use, in this case the current directory, therefore ./default.nix.
The -i stands for installation.
The -A is the same as above for nix-build.

We reproduced the very basic behavior of nixpkgs.

The inputs pattern


After a long preparation, we finally arrived. I know you have a big doubt in this moment. It's about the hello.nix and graphviz.nix. They are very, very dependent on nixpkgs:
  • First big problem: they import nixpkgs directly. In autotools.nix instead we pass nixpkgs as an argument. That's a much better approach.
  • Second problem: what if we want a variant of graphviz without libgd support?
  • Third problem: what if we want to test graphviz with a particular libgd version?
The current answer to the above questions is: change the expression to match your needs (or change the callee to match your needs).
With the inputs pattern, we choose to give another answer: let the user change the inputs of the expression (or change the caller to pass different inputs).

By inputs of an expression, we refer to the set of derivations needed to build that expression. In this case:
  • mkDerivation from autotools. Recall that mkDerivation has an implicit dependency on the toolchain.
  • libgd and its dependencies.
The src is also an input but it's pointless to change the source from the caller. For version bumps, in nixpkgs we prefer to write another expression (e.g. because patches are needed or different inputs are needed).

Goal: make package expressions independent of the repository.

How do we achieve that? The answer is simple: use functions to declare inputs for a derivation. Doing it for graphviz.nix, will make the derivation independent of the repository and customizable:
{ mkDerivation, gdSupport ? true, gd, fontconfig, libjpeg, bzip2 }:

mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
buildInputs = if gdSupport then [ gd fontconfig libjpeg bzip2 ] else [];
}
I recall that "{...}: ..." is the syntax for defining functions accepting an attribute set as argument.

We made gd and its dependencies optional. If gdSupport is true (by default), we will fill buildInputs and thus graphviz will be built with gd support, otherwise it won't.
Now back to default.nix:
let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in with pkgs; {
hello = import ./hello.nix { inherit mkDerivation; };
graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; };
graphvizCore = import ./graphviz.nix {
inherit mkDerivation gd fontconfig libjpeg bzip2;
gdSupport = false;
};
}
So we factorized the import of nixpkgs and mkDerivation, and also added a variant of graphviz with gd support disabled. The result is that both hello.nix (exercise for the reader) and graphviz.nix are independent of the repository and customizable by passing specific inputs.
If you wanted to build graphviz with a specific version of gd, it would suffice to pass gd = ...;.
If you wanted to change the toolchain, you may pass a different mkDerivation function.

Clearing up the syntax:
  • In the end we return an attribute set from default.nix. With "let" we define some local variables.
  • We bring pkgs into the scope when defining the packages set, which is very convenient instead of typing everytime "pkgs".
  • We import hello.nix and graphviz.nix, which will return a function, and call it with a set of inputs to get back the derivation.
  • The "inherit x" syntax is equivalent to "x = x". So "inherit gd" here, combined to the above "with pkgs;" is equivalent to "x = pkgs.gd".
You can find the whole repository at the pill 12 gist.

Conclusion


The "inputs" pattern allows our expressions to be easily customizable through a set of arguments. These arguments could be flags, derivations, or whatelse. Our package expressions are functions, don't think there's any magic in there.

It also makes the expressions independent of the repository. Given that all the needed information is passed through arguments, it is possible to use that expression in any other context.

Next pill


...we will talk about the "callPackage" design pattern. It is tedious to specify the names of the inputs twice, once in the top-level default.nix, and once in the package expression. With callPackage, we will implicitly pass the necessary inputs from the top-level expression.

Pill 13 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Nix pill 11: the garbage collector

Welcome to the 11th Nix pill. In the previous 10th pill we managed to obtain a self-contained environment for developing a project. The concept is that whereas nix-build is able to build a derivation in isolation, nix-shell is able to drop us...

Nix pill 10: developing with nix-shell

Welcome to the 10th Nix pill. In the previous 9th pill we have seen one of the powerful features of nix, automatic discovery of runtime dependencies and finalized the GNU hello world package.In return from vacation, we want to hack a little t...