Switching to NixOS from Debian

Background

When I first started running Linux as a desktop OS at home, I picked Debian; I think it had something to do with liking the focus on stability and the community oriented nature of the project. The stability aspect really worked well for me and I encountered very few issues with software breaking in general. In-place upgrades were quick and painless.

Sometimes I did want to have the latest version of an application; usually with music production software like Guitarix or Ardour. I ended up building these by hand and installing into an isolated prefix. There is probably a better way to get newer versions on Debian but I always felt uncomfortable with adding lots of extra apt repositories and fighting with conflicting packages.

At some point, I decided to try NixOS on my laptop while keeping Debian on my main desktop. I initially found the Nix language pretty confusing but I really did like the whole approach of building the system from a configuration, how easy it was to rollback, and the ability to quickly and temporarily install a program to try it with nix-shell -p <package>. I started using Home Manager to manage my user profile with the goal of maintaining a git repository with my configuration, something I've never managed to make myself do with just dotfiles. NixOS was also very stable for me on the release branches and easy to perform major version upgrades.

With the success on my laptop, which is less frequently used, I decided to install the Nix package manager on my Debian desktop. That gave me easy access to up to date software while not having to dive all the way into NixOS and rebuild my main machine. There were some challenges with audio libraries fighting with the system version but I was ultimately able to work through those. That setup worked fine for a long time, but once the Debian version I was on moved to old-stable, I decided it was time to move to NixOS.

Preparing to Migrate

If I am being honest, the main reason it took me so long to move to NixOS was that I was worried about messing up my machine and, more importantly, losing data. It is a bit embarrassing to say, but I did not have good backup hygiene. It was something I thought/worried about a lot, but never sat down and committed myself to figure out a good plan. I decided, however, that solid backups was a prerequisite for me to move to NixOS.

My desktop has a 3 disk RAID array where I was backing up my data. There was definitely some stuff on that volume that was only there, but a lot of it was mirrored copies of what I had on my main drive. To get to a better place, my first priority was to find a backup program that allowed historical snapshots. Then, I needed a way to have an off-site backup.

Backups With Restic

I ended up picking Restic to manage backups for reasons that I do not entirely remember. The basics of managing repositories and creating snapshots are pretty straight forward. I ended up just creating manifest files that listed all the directories and passing that in as the --files-from argument. Simple.

$ restic backup --files-from backup-manifest -r /backup/restic

Off-site With Rclone

One of the reasons I did like Restic was that it had S3 repository back-end. It also seemed possible to simply sync a backup to an S3 compatible storage provider using a sync tool (e.g. aws s3 sync). After reviewing some of the options and some of the warnings about copying between repositories, I decided that simply mirroring the local repository was a better option. Again, my intent was to use S3 but when looking at the actual cost and realizing that I already had a painfully underutilized Dropbox subscription that had more than enough space, I decided that would be a better option.

To sync my Restic backup to Dropbox, I found Rclone which is kind of like rsync, but for all sorts of different back-ends. Setting up the Dropbox back-end was straight forward and then all that was left was running a sync and waiting days as hundreds of gigabytes were uploaded on my slow asymmetric residential cable internet connection.

rclone sync /backup/restic dropbox:restic

Installing NixOS

Now that I had all my files backed up, including off-site, I finally felt like I was ready to move. Since I already had a Nix install, I decided to attempt installing from within Debian and using the "lustrate" process. That let me have the manual up in the browser while following the process.

The NixOS installation process does not prepare any storage devices. That was great for me because I wanted to keep things as they were. Running nixos-generate-config gave me a reasonable looking basic configuration which I made a few tweaks and then followed the rest of the process as documented in the manual.

A Failed Install

After completing the steps in the manual, I crossed my fingers and rebooted. At first it looked successful: grub came up and then NixOS started the phase 1 boot. But then it all went sideways as one of the volumes in my configuration could not be found. Attempting to ignore and boot anyway resulted in a kernel panic. The volume in questioned turned out to be my backup RAID array. I had waffled about whether to start with that in the configuration or not and it turned out to be a mistake.

In order to fix things, I booted from a NixOS USB stick. However, having the old Debian install still partially in place (lustrate had yet to run) messed with nixos-rebuild. For example, the /etc/resolv.conf file broke DNS resolution. In the end, I decided to just create /debian folder and move all of the old system directories into it. That made nixos-rebuild happy as it was no longer dealing with any of the left over Debian files. I commented out my backup volume from my configuration rebuilt and rebooted. This time the system came up.

IOMMU Issue

During boot I was greeted with a somewhat long delay while a pile of IOMMU errors were logged. This system is an old AMD FX-8350 with a Gigabyte 990FXA-UD3 board. This particular issue rang a bell as I did have it when I installed Debian originally. Scrounging around in the old /debian files, I was able to find the kernel command line in my grub configuration and I had added amd_iommu=on and iommu=pt. Finding some other references to that online, I added them to my kernel parameters:

boot.kernelParams = [
  "amd_iommu=on"
  "iommu=pt"
];

After rebuilding that issue was resolved.

Backup Volume Round 2

Now that the system was up and functional I had to figure out why my backup volume caused issues during boot. The backup volume has this device structure:

Physical Disks
|
+ MD Device (LVM Physical Volume)
  |
  + LVM Volume Group
    |
    + LUKS

When inspecting block devices, the MD device was missing. I needed to enable swraid in my NixOS configuration:

boot.swraid = {
  enable = true;
  mdadmConf = ''
    MAILADDR root
  '';
};

Now the MD device was showing up and the LVM volume group was also present. I was able to manually open the LUKS volume and mount it. I went back and uncommented the LUKS device and file system in my configuration. The key is on the encrypted root volume which is unlocked during boot by a passphrase.

boot.initrd.luks.devices.backup_crypt = {
  device = "/dev/mapper/backup--vg-backup";
  keyFile = "/etc/backup-key";
};

After I rebooted, I hit the same error as I had during initial setup. I did not spend too much time trying to see if that approach could work. It seems like the DM device was not being set up during boot as the error was that the LVM VG was not found. On my Debian setup, I had used crypttab to configure encrypted volumes, so I abandoned trying to get it to work during the initial phase and went with crypttab:

environment.etc.crypttab = {
  mode = "0600";
  text = ''
    backup_crypt /dev/mapper/backup--vg-backup /etc/backup-key
  '';
};

That approached worked right away and the volume was available once the system booted.

PipeWire and Firewire

One of the things I use my desktop for is playing guitar and music production. My audio interface is an old MOTU UltraLite MK3 that I bought somewhere around 2008. It has never been great in Linux as the drivers are not terribly stable (the screen going into power save and then back on again really messes with it). But it works, does what I need, and as long as I have a system that supports Firewire, I cannot see a reason to replace it.

I was using Jack on Debian but decided that I needed to get with the times and try PipeWire. I added PipeWire to my configuration and enabled Jack support. However, I could not get my interface to show up. Eventually, I was able to determine that the kernel module was not loading. modprobe firewire-ohci did the trick and I was able to see the device in PipeWire and launch my Jack-based programs and route through my interface.

I limped along initially using modprobe to load the driver but eventually I wanted to figure out why it was not available after boot. I tried adding it to my boot modules but nothing. Eventually, I went and looked at /etc/modproe.d and found that the nixos.conf file had it blacklisted. It took a bit of digging but eventually I figured out the blacklist is automatically enabled when using LUKS to mitigate direct memory access (DMA) vulnerabilities that let someone with physical access extract your encryption key. This is controlled by NixOS option boot.initrd.luks.mitigateDMAAttacks. I contemplated disabling it, but decided to just use modprobe to load and unload the kernel module when I actually needed to use the interface. Security.

The next problem I had with PipeWire was latency; even when I set the buffer size low there was a lot of delay. The fix turned out to be setting my interface into "Pro Audio" mode. I am sure there is a command to set that, but I just used pwvucontrol to toggle it and the setting has stuck.

My experience with PipeWire has been mixed. Having one system that integrates with all the other audio systems is great and it does seem to work really well. However, I was having a lot more drop outs and overruns than I did previously when running with buffer sizes of 128 and 256 and working in Ardour. One difference here is that PipeWire is using ALSA and its drivers for my interface, where I was using FFADO with Jack directly before. I still want to investigate this further and try and resolve it because using PipeWire in the long run would be better.

Jack

Getting Jack as a standalone system (i.e. not stubbed by PipeWire) requires disabling Jack in the PipeWire NixOS options (services.pipewire.jack.enable = false). Once that was switched I was having errors with Jack using real-time priority. On many Linux distributions, an audio user group is created that has limits set by PAM during login to permit using real-time priority. NixOS does not do this by default. While there was an audio group, the limits were not specified. This is straight forward to do:

security.pam.loginLimits = [
  {
    domain = "@audio";
    type = "-";
    item = "rtprio";
    value = "95";
  }
  {
    domain = "@audio";
    type = "-";
    item = "memlock";
    value = "unlimited";
  }
];

Adding memlock also avoids an warning that Ardour would raise during startup. I eventually found the musnix module that sets a lot of these configurations for you but I have not yet tried it.

Summing it Up

In the end, this definitely took a lot more work and frustration than I was expecting. I tend to think of that toil as being worth it overall because I certainly learned something. The really nice aspect of all this is that now the configuration is all in code and in a git repository. In contrast, I had to fix some of the same things in my Debian install but those configurations grew organically over time and are scattered across the whole install. I much prefer the Nix approach. After I had the basics working on the system, I refactored my configuration into multi-host flake setup that I can use across my machines.