Fearless Tinkering with Nix

This is the second installment of Fearless Tinkering is Functional and can be read standalone:

1. Fearless Tinkering is Functional
2. Fearless Tinkering with Nix 👈
3. Fearless Tinkering with Haskell
4. Fearless Tinkering is Algebraic
5. Fearless Functional Future

Fearless in Practice

Beyond the right tool for the job, it is the right values for the job… and then the right software for the values.

• Bryan Cantrill @ Node Summit 2017, Platform as a Reflection of Values

Programmers make mistakes. As such, our tools should aspire not to punish us for unfamiliarity or occasional incorrectness. Too often, users stumble into a tool's legitimate accidental complexity, only to be chided for “holding it wrong” as if they should feel guilty for letting the tool paint them into a corner. Of course, tools that overreact to surface considerations can be just as problematic. No one wants an overzealous tool that “helpfully” enacts what hasn’t been asked for. Instead, tools should focus on preventing unrecoverable choices from being made too casually or too late. When a situation calls for an action with serious implications, our tools should help us feel the weight of our request. Uniformly presenting every possibility will mislead users into thinking that every action is reasonable or fallible. Tools should immediately report suspected incorrectness. Discovering mistakes while constructing a new home is much less painful than making the same discoveries after a family has moved into it.

Approaches that meet each critique with an independent solution are prone to local maxima. Breakthroughs require recognition of the systemic complexities underpinning individual occurrences. Ideally, tools should proactively seek out their fundamental weaknesses rather than waiting to reactively patch ailments.¹ Hence, selecting tools by their structural guarantees is about using one's values to elect the ways in which their software can’t go wrong.

In this and upcoming posts, I’ll introduce several functional technologies and illustrate how their structural properties enable fearless tinkering. These alone won’t do functional ecosystems justice.² Even amongst the tools mentioned, there are plenty of assurances I won't cover. Conversely, I won't spend much time on their structural deficiencies or notable footguns³. Much more important than any specific technology are the ideas that underlie them. Calling out the need for safe approaches to common pain points will help future technologies incorporate the relevant structural guarantees.

Nix & NixOS

The entropy of modern software is staggering. Programs inside programs directing separate programs exist, with each layer containing its own digital metropolis. Whether it's the shear connectors in the bridges, building codes of the plumbing networks, or the newest floor plan atop a towering skyscraper, every software artifact reflects a choice made by someone somewhere. The feats of modern software could not be made if every virtual wheel needed reinventing. A global and communal project of shared decisions is harnessed to base, tweak, and publish new constructions.

Interacting with outside software remains a tremendous challenge despite how foundational accessing external developments is. Every piece of software has its own installation manual and distribution strategy. Attaining pre-built infrastructure or the materials to create one’s own means traversing complicated and intersecting digital supply chains. Brittle runbooks, non-reproducible scripts, and imperative packaging tools that freely step on each other’s work lead to uniquely broken and divergent systems. Fundamental issues like build-time variability, unspecified dependencies, and implicit mutation of shared libraries can turn one-off upgrades into intractable archeological treasure hunts. When stuck in dependency hell, a teammate voicing that some software “works on their machine” offers more dread than helpfulness.

Nix attempts to solve these problems by introducing a purely functional DevOps ecosystem. By representing all software components in an immutable graph, Nix can offer rich queries about one’s system and defang what were previously panic-inducing deployments. With a declarative language for reproducible system configurations, installing software simply becomes the act of writing a Nix expression that describes a target system with that package installed. This process is valid for applying parameter configurations, upgrades, uninstallations, patches, or any deployment action. Instead of divining sequential migration paths, developers and operators can focus solely on what they want their end system to look like. The evolution of traditional infrastructure is contrasted with Nix below.

Road to Reproducibility

For many developers, infrastructure is an uncomfortable area. Whether it’s the state of a cluster on us-east-1 or a configuration outside one’s project repository, reasoning about action at a distance can be treacherous. Further, the stakes are often high when managing software deployments. A tangled web of mutual, implicit dependencies means modifying one’s system could lead to invalid and unrecoverable states. Tolerating a “working” yet unsatisfactory environment is frequently preferable to the risks of transitioning to something better. As experimentation with alternatives is difficult, iterating is normally restricted to when one’s hand is forced, or the alternative has already been intimately understood. Even the modern practice of automated upgrades for existing dependencies is fraught. Tools for managing remote infrastructure tend to remain unfamiliar as the consequences of trying them out might yield an expensive $CLOUD_PROVIDER bill.

A horrible culmination of these frictions is found in operational firefighting. Operators are tasked with debugging a live system whose interface, tools, and symptoms are typically only practiced against when problems occur. Even in organizations with significant platform investments, the exploratory surface of test deployments rarely reflects the time and space pollution built up across all contributing local developer and remote production environments.

Application programming, in contrast, can give developers a sense of power and ease. Assuming a stable working environment, the behavior and outputs of a local program are far more observable. Equipped with a test harness, REPL, and breakpoints, an interactive view of one’s program, including local bindings and available operations, can be explored. An application programmer can wield static analyzers, linters, IDE feedback, and all their most familiar tools to iteratively coax their programs into the desired shape. The ultimate execution of their developments can be safely gated behind various evaluation stages like continuous integration, peer review, staging deployments, and quality assurance.

The extent to which work is disposable is the extent to which work is safe to do. While it may mean tremendous pain in the short term, application programmers can usually throw their hands up and start over from scratch when a local problem arises. If developers tie their local project repository into a knot, they can blow it away and pull down a fresh copy from version control. During development, application programmers regularly break their domain in ways that would be intolerable anywhere else. Though the behavior of real-world programs are inseparable from their execution environments, application programmers can approximate control over time and space by fixing their efforts against their own environment.

Can the boons of application programming extend to infrastructure? Containers and Infrastructure as Code (IaC) are frequently touted as solutions, yet their standard implementations fall short. Containers and microVMs offer various levels of isolation; however, these "encapsulated" environments lack innate compositionality and reproducibility. Docker, for instance, is "repeatable, but not reproducible." Delineating the steps for constructing an image and subsequently executing those steps won't guarantee a uniform result. At most, a Dockerfile in version control can be nominally linked to an image within a container registry. Although containers can mitigate configuration drift when the same image is employed, they don't ensure consistency when creating a new image or even rebuilding an existing one. The mutable nature of prevalent base images propels users onto unstable grounds and introduces numerous compliance issues. Even more troubling is the fact that standard auditing methods, such as container vulnerability scanners, are easily thwarted. Containers merely shift around the complexities of imperativity rather than discharging them. Instead of "works on my machine," containers settle for "works with my image."

Similarly, IaC solutions that advertise themselves as "immutable infrastructure", yet don't offer compositionality or reproducibility, leave a lot to be desired. These tools introduce an inconsistent experience, where environments are delivered declaratively but developed imperatively. Masking a leaky abstraction like mutation with a 'declarative' interface obscures pathologies and undermines the original appeal: simplicity.

Meanwhile, Nix has developed to fill these gaps. Nix can fit neatly within existing infrastructure, letting engineers recover compositionality and reproducibility while using IaC, containers, and other solutions. Nix can also subsume various existing tools with alternatives like NixOS, Home-Manager, Cachix, Hydra, and deploy-rs. This versatility allows Nix to be introduced piecemeal into projects and organizations.

Nix: A Minimal Primer

Move fast and save things.

• Mark Functorberg on immutability, probably

In the Nix language, everything is an expression. This includes Nix's central data structure, the derivation. Derivations can be thought of as recipes for building software components. These recipes include stable references to the nested build plans of their direct dependencies. A crucial part of keeping Nix "purely functional" is the separation of evaluation and execution. This distinction is shown by the two steps that make up a Nix build:

1. Instantiation - pure evaluation of Nix expression for assembling a derivation
2. Realisation - execution of derivation's instructions + addition of outputted component to the Nix Store

The Nix Store is an append-only graph database that stores software components in the host's file system (typically under /nix/store). Every package, service, and configuration is stored in an immutable way, meaning it cannot be modified after creation. Instead, changes result in new component versions, leaving the previous ones intact.

The location of a component in the Nix Store is determined by the hash of its derivation. As a derivation contains references to its exact inputs, Nix can ensure that a given component is built the same way every time. This property is called input-addressable and is how Nix works by default. For example, a foo component whose derivation hashes to 9fjirpbq will be stored at /nix/store/9fjirpbq-foo. Any change to a transitive dependency in the Nix expression defining foo will result in a different derivation and, thus, a new component being written to the Nix Store (ex. /nix/store/3wby0ym3-foo).

The Nix literature defines "correct deployments" as software deployments that "given identical inputs, should behave the same on an end-user machine as on the developer machine." This is a useful albeit weaker notion of reproducibility than bit-for-bit reproducibility. While Nix can ensure builds are executed with the same steps and inputs, it can't guarantee a deterministic output if the underlying package requires being built with non-deterministic tooling. Rather than storing components by the hash of their build plan, support is progressing for storing components by the hash of their build outputs. This property is called content-addressable and enables reduced recompilation via early cutoff. For obvious reasons, bit-for-bit reproducibility is useful for content-addressable builds.

Nix offers a rich set of structural properties and shares a lot in common with other Merkle structures like git. While useful on their own, Nix's properties provide more significant benefits together than the sum of their parts.

Fearless Environments

The world is all that is the case. A computation is any action taken by a programmable device. An environment is the set of all past and present worlds that a computation can observe. An effect is a computation that depends on or changes the program’s environment. We run programs to produce effects in the world around us.

• Patrick Thomson @ Strange Loop 2019, definitions from "Building Haskell Programs with Fused Effects"

Nix's package manager is transactional. Package state is isolated, state transitions are atomic, and these state transitions are consistent. Builds are uniquely addressed, immutable, reproducibly resourced, and always safe to interrupt, restart, or run in parallel. As elaborated below, these essential properties form transactional boundaries within which programmers can fearlessly develop environments.

Atomic

Whether installing a new package or upgrading one's system, changes through Nix happen all at once or not at all. Even if a deployment operation is interrupted, users are never left in a partially updated state. If a completed upgrade causes issues, one can immediately return to their previous configuration. As a result, iterating on one's environment can be done without commitment. Users can trial packages using ephemeral shells that don't pollute their environments. Build-time dependencies and binaries for one-off commands can be pruned beyond their usage. The combination of rollbacks and version control lets one recreate their environment from any point in the past. If one no longer needs a component, they can confidently remove it without fear of changing their mind. This results in less package hoarding and more minimal configurations.

Isolated

As each package is stored in a unique path, different versions or variants can coexist without conflict. Users can create and modify multiple developer environments for multiple projects without breaking one another. One could define a completely isolated developer environment for a project containing all tooling, like local commands, fully-configured editors, local databases, and more. Alternatively, one could factor general system configuration into a top-level repository, allowing other projects to inherit configurations while remaining explicit.

Immutable

Nix uses top-level views called profiles to present immutable environments to users. These views provide a native user experience through symlinks to store paths. For example, a user with a Python package in their profile can simply type python into their shell instead of a fully qualified path like /nix/store/7fkcip5k-python3-3.11.3/bin/python. While these top-level views exhibit a form of mutability, modifications can be reliably reverted. Changing a user's environment will result in a new Nix Store component that is then pointed to by the user's current view. Environments that share the same lineage are called 'generations', and can be 'rolled back' using Nix's CLI to recover a previous profile.

Fearless Sharing

When you share code with the world, how are they supposed to build it? Are they sure that they are running the same code that you, the author and builder, intended for them to run?

As engineers, we would love for all these to be true: I show up in a README, it's one command to build the project, run it, and it's built in the exact same way that the developer intended. There's nothing hidden about the build process and I can see that it's repeatable. This is transparency and this is what open source is all about.

• Julien Urraca @ Open Source 101 2023, "From Nix to Docker - Everything You Need to Know About Polygot Package Management"

The classic advice to "look both ways before crossing" is extremely applicable to software development. Most programmers are both producers and consumers of software. Without transparent views upstream and downstream, maintenance strategies are tricky and individualized. Untracked mutability creates a big ball of mud that rolls downstream and quickly grows out of control until explicitly confronted.

Sharing is intrinsic to Nix's programming model. Nix allows users to handle internal and external dependency relations in an explicit and uniform manner. Package users can be sure that package authors aren't implicitly offloading upstream problems onto them, while authors can expect the same of their dependencies. As identically-built artifacts appear in identical locations, the sum cost of immutability is negligible. These properties enable programmers to fearlessly contribute, distribute, and inherit software.

Transparent

Open source is critical to most commercial software. As such, supply chain security is an increasingly prevalent concern. Across 1,703 commercial codebases scanned in Synopsys' 2023 OSSRA Report:

• 76% of code in codebases came from open source
• 54% of codebases had license conflicts
• 89% of codebases contained open-source code more than four years out-of-date
• 48% of codebases contained high-risk vulnerabilities⁴

It is clear that programmers need better tools for safely working with external software. Programmers need to be able to easily audit their supply chain and get actionable feedback without relying on specialized security researchers. Nix is no silver bullet, but it does provide supply chain transparency. Every artifact in a Nix environment has a traceable origin, a property otherwise known as provenance. Fixed-output derivations, or FODS, make fetching software over the internet more secure by requiring that the received contents have a matching hash to that of the expected contents.

Mutual

Reproducibility enables the reliable distribution of Nix-based infrastructure. Nix enables cloud builds, meaning that native builds are amortized across machines. Whether from Nixpkgs, a CI server, or a developer machine, packages can be built once and shared anywhere. Nix can recover sharing benefits within an infrastructure that isn't Nix-based, like with optimally-layered docker images and system-pinned Bazel builds. Additionally, Nix has world-class support for cross-compilation.

With Nix, code sharing extends beyond the mere transfer of source files. Environments are authenticated by their derivations and encompass the entire context within which the code is intended to run. Nix ensures hermetic and reproducible build inputs through input-addressable storage. Support for content-addressable storage in the future will make components self-authenticating and enable greater de-duplication.

Building from source provides greater trustworthiness, availability, and configurability while incurring greater retrieval, compute, and storage costs. Binary caches are cheaper by comparison but require users that only need what's available. The reproducibility of Nix allows it to blend source and binary deployment models' best aspects while dodging their respective drawbacks.

Persistent

Nix's persistent data model allows components to safely and efficiently share dependencies. When multiple local projects require an identically-built executable, Nix will know to never unnecessarily rebuild or redownload it across projects. As the shared executable is protected within an immutable component store, any changes to a project containing the mutual dependency won't affect the others. Packages can be shared, modified, and upgraded, all without risk of conflict or interference.

Supplementary Examples

    environment.systemPackages = with pkgs; [
-      python310
+      python311
+      firefox
    ];

Auditing Packages

Let's say I wanted to know the transitive dependencies of my local python installation.

A quick way to verify that a package was actually installed via Nix is to trace its location back to the Nix Store:

$ readlink $(which python)
/nix/store/7fkcip5klr0gksrjgdb3xf0wiz89mdak-python3-3.11.3/bin/python

From there, one possible route for gathering dependencies could be to inspect the package's build plan (called a derivation) to find direct dependencies and the build plans of those dependencies:

$ nix derivation show $(which python) | jq '.[].inputDrvs | keys'
[
  "/nix/store/1sh46d9v2n87kabvhnsqqpsrn0l8j9qb-readline-8.2p1.drv",
  "/nix/store/3yzr5644didynr33gmg8ps3pd40i5gsk-ncurses-6.4.drv",
  "/nix/store/4k5amvhhn2ahfj4lnwfmhdb3i6fd3pc8-configd-453.19.drv",
  "/nix/store/5l1qsivyqzvwim32c4lqlw08gqkr1pda-libffi-3.4.4.drv",
  "/nix/store/ah5g0p1vca3izf2z72w7xwgi0gwfr2rz-sqlite-3.41.2.drv",
  "/nix/store/anqxwbxv66qwqwnxmrk5dczlzawan3w4-bash-5.2-p15.drv",
  "/nix/store/b05r3wxalqnkhik6d60xxjbizx01vd32-Python-3.11.3.tar.xz.drv",
  "/nix/store/bd9w0gdpl5h8n97yfgv0i6mdwyqz4lv9-xz-5.4.3.drv",
  "/nix/store/dgy0q5xz9h8yphvn72rrpmdsnvhll5d9-gdbm-1.23.drv",
  "/nix/store/dkx3cs1vqvsafsca08vx681d5jhhsr8v-bzip2-1.0.8.drv",
  "/nix/store/j0xw7lx8bxi6098psd1x9qgqq8k3186d-zlib-1.2.13.drv",
  "/nix/store/kw4n89mqr55jzqpw66sncyvpgs3lscyx-libxcrypt-4.4.33.drv",
  "/nix/store/mbqjy5ff44qsw8bzi7b4q07hnhp5nmy5-mailcap-2.1.53.drv",
  "/nix/store/n92l3i5n4a2z3gy7qhvq6545vvrih0cz-stdenv-darwin.drv",
  "/nix/store/nh3pwqzjjsh9ldvimxswsslrs7vw58kb-nuke-references.drv",
  "/nix/store/vhsawk7n84qsb052923kc6mc8rxzg1c8-python-setup-hook.sh.drv",
  "/nix/store/xxwl7wc8pyf66z89w1s3x0h1mbx1kawc-expat-2.5.0.drv",
  "/nix/store/ymxifdj2b2li33jybh29d5c9v79i403v-openssl-3.0.8.drv",
  "/nix/store/zvh05gmr58z8fkjlnchj5vyd81nbvz23-tzdata-2023c.drv"
]

One could imagine recursively inspecting ancestral build plans to collect transitive dependencies. Fortunately, there are already ways to do this with the Nix CLI:

$ nix path-info -r $(which python) | head
/nix/store/c3d1f6gg0lzfck5cx8yn86s9m5xdbxhg-nghttp2-1.51.0-lib
/nix/store/c834w8pg8ypl9lcaxqgj2bpsxbv518rn-Libsystem-1238.60.2
/nix/store/dr3pb8n9f7lq734ck2sq07r8c1nzs068-brotli-1.0.9-lib
/nix/store/fszqml5r6d35c95c0gf98j977nrlmj8j-zlib-1.2.13
/nix/store/7kc331g8kcpbrkaqyx7rnqd1mnkk08fm-bash-5.2-p15
/nix/store/ghaalrh55chsl4rgw1l74d8ibwzbk2fm-libkrb5-1.20.1
/nix/store/nxjqs8gv8al8j94p34vci1k3yxd5yb2r-openssl-3.0.8
/nix/store/jdqi02l76hzkpbxnjd4yl5y8rjbsjdgi-libssh2-1.10.0
/nix/store/ljsji154hcn08ipa5lnmq64qvvdjkiqi-libiconv-50
/nix/store/792ghrrsff5p51m13a7sf8bpwxc6isdx-libunistring-1.1

The Nix CLI lets us directly query the full dependency graph (called a transitive closure) and visualize it as either a directory tree or a graphviz image:

$ nix-store --query $(which python) --tree | head
/nix/store/7fkcip5klr0gksrjgdb3xf0wiz89mdak-python3-3.11.3
├───/nix/store/1ds96f193bdf7f3kz295rwf66q90vi71-swift-corefoundation-unstable-2018-09-14
│   ├───/nix/store/42j0x4w0g5jpnq0lhix73hqnff865ahl-curl-8.0.1
│   │   ├───/nix/store/c3d1f6gg0lzfck5cx8yn86s9m5xdbxhg-nghttp2-1.51.0-lib
│   │   │   └───/nix/store/c3d1f6gg0lzfck5cx8yn86s9m5xdbxhg-nghttp2-1.51.0-lib [...]
│   │   ├───/nix/store/c834w8pg8ypl9lcaxqgj2bpsxbv518rn-Libsystem-1238.60.2
│   │   │   └───/nix/store/c834w8pg8ypl9lcaxqgj2bpsxbv518rn-Libsystem-1238.60.2 [...]
│   │   ├───/nix/store/dr3pb8n9f7lq734ck2sq07r8c1nzs068-brotli-1.0.9-lib
│   │   │   └───/nix/store/dr3pb8n9f7lq734ck2sq07r8c1nzs068-brotli-1.0.9-lib [...]
│   │   ├───/nix/store/fszqml5r6d35c95c0gf98j977nrlmj8j-zlib-1.2.13

$ nix-store --query $(which python) --graph | nix-shell --packages graphviz --command 'dot -Tpng > python_deps.png'

Alternatively, we could interactively inspect these dependencies with the nix-tree derivation browser.

$ nix-tree $(which python)

Let's say I wanted to distribute this dependency information to others in a commonly understood format. Like ingredient labels on food packaging, SBOMs are an increasingly standard method of providing supply chain transparency for software. SBOMs are a natural fit for Nix as all required provenance is already present in derivations. Furthermore, Nix provides a great foundation for wider supply chain security frameworks like SLSA.

Below, I'll use sbomnix and a temporary visidata installation to tabularly inspect the ingredients of my python package:

$ sbomnix $(which python)
WARNING  Command line argument '--meta' missing: SBOM will not include license information (see '--help' for more details)
INFO     Loading runtime dependencies referenced by '/nix/store/7fkcip5klr0gksrjgdb3xf0wiz89mdak-python3-3.11.3/bin/python3.11'
INFO     Wrote: sbom.cdx.json
INFO     Wrote: sbom.spdx.json
INFO     Wrote: sbom.csv

$ nix-shell --packages visidata --command 'vd sbom.csv'

As the entire build specification for my python installation exists in the Nix Store, tools like vulnix can be used to audit it against the NIST National Vulnerability Database. Moreover, vulnix --profile can be used to validate all packages in my user environment while vulnix --system validates all packages in the Nix Store.

$ vulnix $(which python) | head
9 derivations with active advisories

------------------------------------------------------------------------
binutils-2.40

/nix/store/cwpwjv8f2pxfkc52m3ym78dkap6whl4y-binutils-2.40.drv
CVE                                                CVSSv3
https://nvd.nist.gov/vuln/detail/CVE-2023-1972     6.5

------------------------------------------------------------------------

Additionally, vulnxscan supports SBOM-based vulnerability scans against OSV and Grype.

pkgs.dockerTools.buildImage {
  name = "hello";
  tag = "latest";
  copyToRoot = pkgs.buildEnv {
    name = "image-root";
    paths = [ pkgs.hello ];
    pathsToLink = [ "/bin" ];
  };

  config.Cmd = [ "/bin/hello" ];
}

To Be Continued...

That's all for now! In the next part, we'll dive into Haskell before discussing non-functional technologies that exploit their algebra.

✌️

Footnotes