Blog

Nix pill 7: a working derivation

Welcome to the seventh Nix pill. In the previous sixth pill we introduced the notion of derivation in the Nix language. How to define a raw derivation and how to (try) to build it.In this post, we will continue along the path, by creating a d…

Nix pill 6: our first derivation

Welcome to the sixth Nix pill. In the previous fifth pill we introduced functions and imports. Functions and imports are very simple concepts that allows for building complex abstractions and composition of modules to build a flexible Nix system.

In this post we finally arrived to writing a derivation. Derivations are the building blocks of a Nix system, from a file system view point. The Nix language is used to describe such derivations.

I remind you how to enter the Nix environment: source ~/.nix-profile/etc/profile.d/nix.sh

The derivation function

The derivation built-in function is used to create derivations. I invite you to read the link in the Nix manual about the derivation built-in. A derivation from a Nix language view point is simply a set, with some attributes. Therefore you can pass the derivation around with variables like anything else.
That’s where the real power comes in. 
The derivation function receives a set as first argument. This set requires at least the following three attributes:
  • name: the name of the derivation. In the nix store the format is hash-name, that’s the name.
  • system: is the name of the system in which the derivation can be built. For example, x86_64-linux.
  • builder: it is the binary program that builds the derivation.
First of all, what’s the name of our system as seen by nix?
nix-repl> builtins.currentSystem
"x86_64-linux"
Let’s try to fake the name of the system:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> d
«derivation /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv»
Oh oh, what’s that? Did it build the derivation? No it didn’t, but it did create the .drv file. nix-repl does not build derivations unless you tell to do so.

Digression about .drv files

What’s that .drv file? It is the specification of how to build the derivation, without all the Nix language fuzz.
Before continuing, some analogies with the C language:
  • .nix files are like .c files
  • .drv files are intermediate files like .o files. The .drv describes how to build a derivation, it’s the bare minimum information.
  • out paths are then the product of the build
Both drv paths and out paths are stored in the nix store as you can see.
What’s in that .drv file? You can read it, but it’s better to pretty print it.
$ nix-env -i strategoxt
If you feel the above command being too slow (ignore the meaning of the command below, just do it):
$ nix-env -iA strategoPackages018.strategoxt -f '<nixpkgs>'
The installed pp-aterm program can be used to pretty print .drv files:
$ pp-aterm -i /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
Derive(
  [("out", "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname", "", "")]
, []
, []
, "mysystem"
, "mybuilder"
, []
, [ ("builder", "mybuilder")
  , ("name", "myname")
  , ("out", "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname")
  , ("system", "mysystem")
  ]
Ok we can see there’s an out path, but it does not exist yet. We never told Nix to build it, but we know beforehand where the build output will be. Why?

Think, if Nix ever built the derivation just because we accessed it in Nix, we would have to wait a long time if it was, say, Firefox. That’s why Nix let us know the path beforehand and keep evaluating the Nix expressions, but it’s still empty because no build was ever made.

Important: the hash of the out path is based solely on the input derivations in the current version of Nix, not on the contents of the build product. It’s possible however to have content-addressable derivations for e.g. tarballs as we’ll see later on.

Many things are empty in that .drv, however I write a summary of the .drv format for you:
  1. The output paths (they can be multiple ones). By default nix creates one out path called “out”.
  2. The list of input derivations. It’s empty because we are not referring to any other derivation. Otherwise, there would a list of other .drv files.
  3. The system and the builder executable (yes, it’s a fake one).
  4. Then a list of environment variables passed to the builder.
That’s it, the minimum necessary information to build our derivation.

Important note: the environment variables passed to the builder are just those you see in the .drv plus some other Nix related configuration (number of cores, temp dir, …). The builder will not inherit any variable from your running shell, otherwise builds would suffer from non-determinism.

Back to our fake derivation

Let’s build our really fake derivation:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> :b d
[...]
these derivations will be built:
  /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
building path(s) `/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname'
error: a `mysystem' is required to build `/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv', but I am a `x86_64-linux'
The :b is a nix-repl specific command to build a derivation. You can see more commands with 😕 .
So in the output you can see that it takes the .drv as information on how to build the derivation. Then it says it’s trying to produce our out path. Finally the error we were waiting for: that derivation can’t be built on our system.
We’re doing the build inside nix-repl, but what if we don’t want to use nix-repl?
You can realise a .drv with:
$ nix-store -r /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
You will get the same output as before.
Let’s fix the system attribute:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = builtins.currentSystem; }
nix-repl> :b d
[...]
build error: invalid file name `mybuilder'
A step forward: of course, that “mybuilder” executable does not really exist. Stop for a moment.

What’s in a derivation set

I find useful to inspect the returned value from the derivation function for you.
First of all, the returned value is a plain set:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> builtins.isAttrs d
true
nix-repl> builtins.attrNames d
[ "all" "builder" "drvAttrs" "drvPath" "name" "out" "outPath" "outputName" "system" "type" ]
You can guess what builtins.isAttrs does, it returns true if the argument is a set. While builtins.attrNames returns a list of keys of the given set. Some kind of reflection, you might say.
Start from drvAttrs:
nix-repl> d.drvAttrs
{ builder = "mybuilder"; name = "myname"; system = "mysystem"; }
That’s basically the input we gave to the derivation function. Also d.name, d.system and d.builder attributes are straight the ones we gave as input.
nix-repl> (d == d.out)
true
So out is just the derivation itself, it seems weird but the reason is that we only have one output from the derivation. That’s also the reason why d.all is a singleton. We’ll see multiple outputs later.
The d.drvPath is the path of the .drv file: /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv .
Something interesting is the type attribute. It’s “derivation”. Nix does add a little of magic to sets with type derivation, but not that much. To let you understand, you can create yourself a set with that type, it’s a simple set:
nix-repl> { type = "derivation"; }
«derivation ???»
Of course it has no other information, so Nix doesn’t know what to say 🙂 But you get it, the type = “derivation” is just a convention for Nix and for us to understand the set is a derivation.

When writing packages, we are interested in the outputs. The other metadata is needed for Nix to know how to create the drv path and the out path.
The outPath attribute is the build path in the nix store: /nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname .

Referring to other derivations

Just like dependencies in other package managers, how do we refer to other packages? How do we refer to other derivations in terms of files on the disk?
We use the outPath. The outPath tells where the files are of that derivation. To make it more convenient, Nix is able to do a conversion from a derivation set to a string.
nix-repl> d.outPath
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
nix-repl> builtins.toString d
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
Nix does the “set to string conversion” as long as there is the outPath attribute (much like a toString method in other languages):
nix-repl> builtins.toString { outPath = "foo"; }
"foo"
nix-repl> builtins.toString { a = "b"; }
error: cannot coerce a set to a string, at (string):1:1
Say we want to use binaries from coreutils (ignore the nixpkgs etc.):
nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> coreutils
«derivation /nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv»
nix-repl> builtins.toString coreutils
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21"
Apart the nixpkgs stuff, just think we added to the scope a series of variables. One of them is coreutils. It is the derivation of the coreutils package you all know of from other Linux distributions. 
It contains basic binaries for GNU/Linux systems (you may have multiple derivations of coreutils in the nix store, no worries):
$ ls /nix/store/*coreutils*/bin
[...]
I remind you, inside strings it’s possible to interpolate Nix expressions with ${…}:
nix-repl> "${d}"
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
nix-repl> "${coreutils}"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21"
That’s very convenient, because then we could refer to e.g. the bin/true binary like this:
nix-repl> "${coreutils}/bin/true"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true"

An almost working derivation

In the previous attempt we used a fake builder, “mybuilder” which obviously does not exist. But we can use for example bin/true, which always exits with 0 (success).
nix-repl> :l <nixpkgs>
nix-repl> d = derivation { name = "myname"; builder = "${coreutils}/bin/true"; system = builtins.currentSystem; }
nix-repl> :b d
[...]
builder for `/nix/store/d4xczdij7xazjfm5kn4nmphx63mpv676-myname.drv' failed to produce output path `/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-myname'
Another step forward, it executed the builder (bin/true), but the builder did not create the out path of course, it just exited with 0.
Obvious note: everytime we change the derivation, a new hash is created.
Let’s examine the new .drv now that we referred to another derivation:
$ pp-aterm -i /nix/store/d4xczdij7xazjfm5kn4nmphx63mpv676-myname.drv
Derive(
  [(“out”, “/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-myname“, “”, “”)]
, [(“/nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv“, [“out”])]
, []
, “x86_64-linux”
, “/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true”
, []
, [ (“builder”, “/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true”)
  , (“name”, “myname”)
  , (“out”, “/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-myname“)
  , (“system”, “x86_64-linux”)
  ]
Aha! Nix added a dependency to our myname.drv, it’s the coreutils.drv. Before doing our build, Nix should build the coreutils.drv. But since coreutils is already in our nix store, no build is needed, 
it’s already there with out path /nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21 .

When is the derivation built

Nix does not build derivations during evaluation of Nix expressions. In fact, that’s why we have to do “:b drv” in nix-repl, or use nix-store -r in the first place.

An important separation is made in Nix:

  • Instantiate/Evaluation time: the Nix expression is parsed, interpreted and finally returns a derivation set. During evaluation, you can refer to other derivations because Nix will create .drv files and we will know out paths beforehand. This is achieved with nix-instantiate.
  • Realise/Build time: the .drv from the derivation set is built, first building .drv inputs (build dependencies). This is achieved with nix-store -r.
Think of it as of compile time and link time like with C/C++ projects. You first compile all source files to object files. Then link object files in a single executable.
In Nix, first the Nix expression (usually in a .nix file) is compiled to .drv, then each .drv is built and the product is installed in the relative out paths.

Conclusion

Is that complicated to create a package for Nix? No it’s not.

We’re walking through the fundamentals of Nix derivations, to understand how they work, how they are represented.

Packaging in Nix is certainly easier than that, but we’re not there yet in this post. More Nix pills are needed.
With the derivation function we provide a set of information on how to build a package, and we get back the information about where the package was built.
Nix converts a set to a string when there’s an outPath, that’s very convenient. With that, it’s easy to refer to other derivations.

When Nix builds a derivation, it first creates a .drv file from a derivation expression, and uses it to build the output. It does so recursively for all the dependencies (inputs). It “executes” the .drv files like a machine. Not much magic after all.

Next pill

…we will finally write our first working derivation. Yes, this post is about “our first derivation”, but I never said it was a working one

Pill 7 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Nix pill 5: functions and imports

Welcome to the fifth Nix pill. In the previous fourth pill we touched the Nix language for a moment. We introduced basic types and values of the Nix language, and basic expressions such as “if“, “with” and “let“. I invite you to re-read about these expressions and play with them in the repl.

Functions help to build reusable components in a big repository like nixpkgs. The Nix manual has a great explanation of functions. Let’s go: pill on one hand, Nix manual on the other hand.
I remind you how to enter the Nix environment: source ~/.nix-profile/etc/profile.d/nix.sh

Nameless and single parameter

Functions are anonymous (lambdas), and only have a single parameter. The syntax is extremely simple. Type the parameter name, then “:“, then the body of the function.
nix-repl> x: x*2
«lambda»
So here we defined a function that takes a parameter x, and returns x*2. The problem is that we cannot use it in any way, because it’s unnamed… joke!
We can store functions in variables.
nix-repl> double = x: x*2
nix-repl> double
«lambda»
nix-repl> double 3
6
As usual, please ignore the special syntax for assignments inside nix-repl.
So, we defined a function x: x*2 that takes one parameter x, and returns x*2. This function is then assigned to the variable double.
Finally we did our first function call: double 3.
Big note: it’s not like many other programming languages where you write double(3). It really is double 3.
In summary: to call a function, name the variable, then space, then the argument. Nothing else to say, it’s as easy as that.

More than one parameter

How do we create a function that accepts more than one parameter? For people not used to functional programming, this may take a while to grasp. Let’s do it step by step.
nix-repl> mul = a: (b: a*b)
nix-repl> mul
«lambda»
nix-repl> mul 3
«lambda»
nix-repl> (mul 3) 4
12
We defined a function that takes the parameter “a“, the body returns another function. This other function takes a parameter “b” and returns a*b.
Therefore, calling “mul 3” returns this kind of function: b: 3*b. In turn, we call the returned function with 4, and get the expected result.

You don’t have to use parenthesis at all, Nix has sane priorities when parsing the code:
nix-repl> mul = a: b: a*b
nix-repl> mul
«lambda»
nix-repl> mul 3
«lambda»
nix-repl> mul 3 4
12
nix-repl> mul (6+7) (8+9)
221
Much more readable, you don’t even notice that functions only receive one argument.
Since the argument is separated by a space, to pass more complex expressions you need parenthesis. In other common languages you would write  mul(6+7, 8+9).
Given that functions have only one parameter, it is straightforward to use partial application:
nix-repl> foo = mul 3
nix-repl> foo 4
12
nix-repl> foo 5
15
We stored the function returned by mul 3 into a variable foo, then reused it.

Arguments set

Now this is a very cool feature of Nix. It is possible to pattern match over a set in the parameter. We write an alternative version of mul = a: b: a*b first by using a set as argument, then using pattern matching.
nix-repl> mul = s: s.a*s.b
nix-repl> mul { a = 3; b = 4; }
12
nix-repl> mul = { a, b }: a*b
nix-repl> mul { a = 3; b = 4; }
12
In the first case we defined a function that accepts a single parameter. We then access attributes “a” and “b” from the given set.
Note how the parenthesis-less syntax for function calls is very elegant in this case, instead of doing mul({ a=3; b=4; }) in other languages.

In the second case we defined an arguments set. It’s like defining a set, except without values. We require that the passed set contains the keys “a” and “b“. Then we can use those “a” and “b” in the function body directly.
nix-repl> mul { a = 3; b = 4; c = 6; }
error: anonymous function at (string):1:2 called with unexpected argument `c', at (string):1:1
nix-repl> mul { a = 3; }
error: anonymous function at (string):1:2 called without required argument `b', at (string):1:1
Only a set with exactly the attributes required by the function is accepted, nothing more, nothing less.

Default and variadic attributes

It is possible to specify default values of attributes in the arguments set:
nix-repl> mul = { a, b ? 2 }: a*b
nix-repl> mul { a = 3; }
6
nix-repl> mul { a = 3; b = 4; }
12
Also you can allow passing more attributes (variadic) than the expected ones:
nix-repl> mul = { a, b, ... }: a*b
nix-repl> mul { a = 3; b = 4; c = 2; }
However, in the function body you cannot access the “c” attribute. The solution is to give a name to the given set with the @-pattern:
nix-repl> mul = s@{ a, b, ... }: a*b*s.c
nix-repl> mul { a = 3; b = 4; c = 2; }
24
That’s it, you give a name to the whole parameter with name@ before the set pattern.
Advantages of using argument sets:
  • Named unordered arguments: you don’t have to remember the order of the arguments.
  • You can pass sets, that adds a whole new layer of flexibility and convenience. 
Disadvantages:
  • Partial application does not work with argument sets. You have to specify the whole attribute set, not part of it.
You may find similarities with Python **kwargs.

Imports

The “import” function is built-in and provides a way to parse a .nix file. The natural approach is to define each component in a .nix file, then compose by importing these files.
Let’s start with the bare metal.
a.nix:
3
b.nix:
4
mul.nix:
a: b: a*b
nix-repl> a = import ./a.nix
nix-repl> b = import ./b.nix
nix-repl> mul = import ./mul.nix
nix-repl> mul a b
12
Yes it’s really that straight. You import a file, and it gets parsed as expression. Note that the scope of the imported file does not inherit the scope of the importer.
test.nix:
x
nix-repl> let x = 5; in import ./test.nix
error: undefined variable `x' at /home/lethal/test.nix:1:1
So how do we pass information to the module? Use functions, like we did with mul.nix .
A more complex example:

test.nix:

{ a, b ? 3, trueMsg ? "yes", falseMsg ? "no" }:
if a > b
then builtins.trace trueMsg true
else builtins.trace falseMsg false
nix-repl> import ./test.nix { a = 5; trueMsg = "ok"; }
trace: ok
true
Explaining:
  • In test.nix we return a function. It accepts a set, with default attributes b, trueMsg and falseMsg.
  • builtins.trace is a built-in function that takes two arguments. The first is the message to display, the second is the value to return. It’s usually used for debugging purposes.
  • Then we import test.nix, and call the function with that set.
So when is the message shown? Only when it’s in need to be evaluated.

Next pill

…we will finally write our first derivation.

Nix pill 6 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Nix pill 4: the basics of the language

Welcome to the fourth Nix pill. In the previous third pill we entered the Nix environment. We installed software as user, managed the profile, switched between generations, and queried the nix store. That’s the very basics of nix administrati…

Nix pill 3: enter the environment

Welcome to the third Nix pill. In the previous second pill we have installed Nix on our running system. Now we can finally play with it a little, things also apply to NixOS users.Enter the environmentIn the previous pill we created a nix user…

Nix pill 2: install on your running system

Welcome to the second Nix pill. In the first pill we briefly described Nix.Now we’ll install Nix on our running system and understand what changed in our system after the installation.Nix installation is as easy as installing any other package. It will…

Nix pill 1: why you should give it a try

IntroductionWelcome to the first post of the Nix in pills series. Nix is a purely functional package manager and deployment system for POSIX. There’s a lot of documentation that describes what Nix, NixOS and related projects are.The purpose of thi…

Debugging and fixing a wmctrl bug in NixOS

Without wmctrl I’m lost. I use it together with xbindkeys so that pressing a combination of keys will raise a window of the desktop (editor, shell, browser, chat, …).It turns out however that in NixOS wmctrl didn’t work well. I had 3 windows opened, …