Automating my home network with Salt

Translations: es - Tags: salt

I'm a lousy sysadmin.

For years, my strategy for managing my machines in my home network has been one of maximum neglect:

  • Avoid distribution upgrades or reinstalls as much as possible, because stuff breaks or loses its configuration.

  • Keep my $HOME intact across upgrades, to preserve my personal configuration files as much as possible, even if it accumulates vast amounts of cruft.

  • Cry when I install a long-running service on my home server, like SMB shares or a music server, because I know it will break when I have to reinstall or upgrade.

About two years ago, I wrote some scripts to automate at least the package installation step, and to set particularly critical configuration files like firewall rules. These scripts helped me lose part of my fear of updates/reinstalls; after running them, I only needed to do a little manual work to get things back in working order. The scripts also made me start to think actively on what I am comfortable with in terms of distro defaults, and what I really need to change after a default installation.

Salt

In my mind there exists this whole universe of scary power tools for large-scale sysadmin work. Of course you would want some automation if you have a server farm to manage. Of course nobody would do this by hand if you had 3000 workstations somewhere. But for my puny home network with one server and two computers? Surely those tools are overkill?

Thankfully that is not so!

My colleague Richard Brown has been talking about the Salt Project for a few years. It is similar to tools for provisioning and configuration management like Ansible or Puppet.

What I have liked about Salt so far is that the documentation is very good, and it has let me translate my little setup into its configuration language while learning some good practices along the way.

I started with the Salt walkthrough, which is pretty nice.

TL;DR: the salt-master is the central box that keeps and distributes configuration to other machines, and those machines are called salt-minions. You write some mostly-declarative YAML in the salt-master, and propagate that configuration to the minions. Salt knows how to "create a user" or "install a package" or "restart a service when a config file changes" without you having to use distro-specific commands.

My home setup and how I want it to be

pambazo - Has a RAID and serves media and stores backups. This is my home server.

tlacoyo - A desktop box, my main workstation.

torta - My laptop, which has seen very little use during the pandemic; in principle it should have an identical setup to my desktop box.

I open the MDNS firewall ports on those three machines so I can use somehost.local to access them directly, without having to set up DNS. Maybe I should learn how to do the latter.

All the machines need my basic configuration files (Emacs, shell prompt), and a few must-have programs (Emacs, Midnight Commander, git, podman).

My workstations need my gitlab/github SSH keys, my Suse VPN keys, some basic infrastructure for development, I need to be able to reinstall and reconstruct them quickly.

All the machines should get the same configuration for the two printers we have at home (a laser that can do two-sided printing, and an inkjet for photos).

My home server of course needs all the configuration for the various services it runs.

I also have a few short-lived virtual machines to test distro images. In those I only need my must-have packages.

Setting up the salt master

My home server works as the "salt master", which is the machine that holds the configuration that should be distributed to other boxes. I am just using the stock salt-master package from openSUSE. The only things I changed in its default configuration were the paths where it looks for configuration, so I can use a git checkout in my home directory instead of /etc/salt. This goes in /etc/salt/master:

# configuration for minions
file_roots:
  base:
    - /home/federico/src/salt-states

# sensitive data to be distributed to minions
pillar_roots:
  base:
    - /home/federico/src/salt-pillar

Setting up minions

It is easy enough to install the salt-minion package and set up its configuration to talk to the salt-master, but I wanted a way to bootstrap that. Salt-bootstrap is exactly that. I can run this on a newly-installed machine:

curl -o bootstrap-salt.sh -L https://bootstrap.saltproject.io
sudo sh bootstrap-salt.sh -w -A 192.168.1.10 -i my-hostname stable

The first line downloads the bootstrap-salt.sh script.

The second line:

  • -w - use the distro's packages for salt-minion, not the upstream ones.

  • -A 192.168.1.10 - the IP address of my salt-master. At this point the machine that is to become a minion doesn't have MDNS ports open yet, so it can't find pambazo.local directly and it needs its IP address.

  • -i my-hostname - Name to give to the minion. Salt lets you have the minion's name different from the hostname, but I want them to be the same. That is, I want my tlacoyo.local machine to be a minion called tlacoyo, etc.

  • stable - Use a stable release of salt-minion, not a development one.

When the script runs, it creates a keypair and asks the salt-master to register its public key. Then, on the salt-master I run this:

salt-key -a my-hostname

This accepts the minion's public key, and it is ready to be configured.

The very basics: set the hostname, open up MDNS in the firewall

I want my hostname to be the same as the minion name. Make it so!

'set hostname to be the same as the minion name':
  network.system:
    - hostname: {{ grains['id'] }}
    - apply_hostname: True
    - retain_settings: True

Salt uses Jinja templates to preprocess its configuration files. One of the variables it makes available is grains, which contains information inherent to each minion: its CPU architecture, amount of memory, OS distribution, and its minion id. Here I am using {{ grains['id'] }} to look up the minion id, and then set it as the hostname.

To set up the firewall for desktop machines, I used YaST and then copied the resulting configuration to Salt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
/etc/firewalld/firewalld.conf:
  file.managed:
    - source: salt://opensuse/desktop-firewalld.conf
    - mode: 600
    - user: root
    - group: root

/etc/firewalld/zones/desktop.xml:
  file.managed:
    - source: salt://opensuse/desktop-firewall-zone.xml
    - mode: 644
    - user: root
    - group: root

firewalld:
  service.running:
    - enable: True
    - watch:
      - file: /etc/firewalld/firewalld.conf
      - file: /etc/firewalld/zones/desktop.xml

file.managed is how Salt lets you copy files to destination machines. In lines 1 to 6, salt://opensuse/desktop-firewalld.conf gets copied to /etc/firewalld/firewalld.conf. The salt:// prefix indicates a path under your location for salt-states; this is the git checkout with Salt's configuration that I mentioned above.

Lines 15 to 20 tell Salt to enable the firewalld service, and to restart it when either of two files change.

Indispensable packages

I cannot live without these:

'indispensable packages':
  pkg.installed:
    - pkgs:
      - emacs
      - git
      - mc
      - ripgrep

pkg.installed takes an array of package names. Here I hard-code the names which those packages have in openSUSE. Salt lets you do all sorts of magic with Jinja and the salt-pillar mechanism to have distro-specific package names, if you have a heterogeneous environment. All my machines are openSUSE, so I don't need to do that.

My username, and personal configuration files

This creates my user:

federico:
  user.present:
    - fullname: Federico Mena Quintero
    - home: /home/federico
    - shell: /bin/bash
    - usergroup: False
    - groups:
      - users

This just copies a few configuration files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
{% set managed_files = [
  [ 'bash_profile',  '.bash_profile',         644 ],
  [ 'bash_logout',   '.bash_logout',          644 ],
  [ 'bashrc',        '.bashrc',               644 ],
  [ 'starship.toml', '.config/starship.toml', 644 ],
  [ 'gitconfig',     '.gitconfig',            644 ],
] %}

{% for file_in_salt, file_in_homedir, mode in managed_files %}

/home/federico/{{ file_in_homedir }}:
  file.managed:
    - source: salt://opensuse/federico-config-files/{{ file_in_salt }}
    - user: federico
    - group: users
    - mode: {{ mode }}

{% endfor %}

I use a Jinja array to define the list of files and their destinations, and a for loop to reduce the amount of typing.

Install some flatpaks

Install the flatpaks I need:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
'Add flathub repository':
  cmd.run:
    - name: flatpak remote-add --if-not-exists --user flathub https://flathub.org/repo/flathub.flatpakrepo
    - runas: federico

{% set flatpaks = [
   'com.github.tchx84.Flatseal',
   'com.microsoft.Teams',
   'net.ankiweb.Anki',
   'org.gnome.Games',
   'org.gnome.Solanum',
   'org.gnome.World.Secrets',
   'org.freac.freac',
   'org.zotero.Zotero',
] %}

install-flatpaks:
  cmd.run:
    - name: flatpak install --user --or-update --assumeyes {{ flatpaks | join(' ') }}
    - runas: federico
    - require:
      - 'Add flathub repository'

Set up one of their configuration files:

/home/federico/.var/app/org.freac.freac/config/.freac/freac.xml:
  file.managed:
    - source: salt://opensuse/federico-config-files/freac.xml
    - user: federico
    - group: users
    - mode: 644
    - makedirs: True

This last file is the configuration for fre:ac, a CD audio ripper. Flatpaks store their configuration files under ~/.var/app/flatpak-name. I configured the app once by hand in its GUI and then copied its configuration file to my salt-states.

Etcetera

The above is not all my setup; obviously things like the home server have a bunch of extra packages and configuration files. However, the patterns above are practically all I need to set up everything else.

How it works in practice

I have a git checkout of my salt-states. When I change them and I want to distribute the new configuration to my machines, I push to the git repository on my home server, and then just run a script that does this:

#!/bin/sh
set -e
cd /home/federico/src/salt-states
git pull
sudo salt '*' state.apply

The salt '*' state.apply causes all machines to get an updated configuration. Salt tells you what changed on each and what stayed the same. The slowest part seems to be updating zypper's package repositories; apart from that, Salt is fast enough for me.

Playing with this for the first time

After setting up the bare salt-master on my home server, I created a virtual machine and immediately registered it as a salt-minion and created a snapshot for it. I wanted to go back to that "just installed, nothing set up" state easily to test my salt-states as if for a new setup. Once I was confident that it worked, I set up salt-minions on my desktop and laptop in exactly the same way. This was very useful!

A process of self-actualization

I have been gradually moving my accumulated configuration cruft from my historical $HOME to Salt. This made me realize that I still had obsolete dotfiles lying around like ~/.red-carpet and ~/.realplayerrc. That software is long gone. My dotfiles are much cleaner now!

I was also able to remove my unused scripts in ~/bin (I still had the one I used to connect to Ximian's SSH tunnel, and the one I used to connect to my university's PPP modem pool), realize which ones I really need, move them to ~/.local/bin and make them managed under Salt.

To clean up that cruft across all machines, I have something like this:

/home/federico/.dotfile-that-is-no-longer-used:
  file.absent

That deletes the file.

In the end I did reach my original goal: I can reinstall a machine, and then get it to a working state with a single command.

This has made me more confident to install cool toys in my home server... like a music server, which brings me endless joy. Look at all this!

One thing that would be nice in Flatpak

Salt lets you know what changed when you apply a configuration, or it can do a dry run where it tells you what will change without actually modifying anything. For example, Salt knows how to query the package database for each distro and tell you if a package needs to be updated, or if an existing config file is different from the one that will be propagated.

It would be nice if the flatpak command-line tool would return this information. In the examples above, I use flatpak remote-add --if-not-exists and flatpak install --or-update to achieve idempotency, but Salt is not able to know what Flatpak actually did; it just runs the commands and returns success or failure.

Some pending things

Some programs keep state along with configuration in the same user-controlled files, and that state gets lost if each run of Salt just overwrites those files:

  • fre:ac, a CD audio ripper, has one of those "tip of the day" windows at startup. On each run, it updates the "number of the last shown tip" somewhere in its configuration for the user. However, since that configuration is in the same file that holds the paths for ripped music and the encoder parameters I want to use, the "tip of the day" gets reset to the beginning every time that Salt rewrites the config file.

  • When you load a theme in Emacs, say, with custom-enabled-theme in custom.el, it stores a checksum of the theme you picked after first confirming that you indeed want to load the theme's code — to prevent malicious themes or something. However, that checksum is stored in another variable in custom.el, so if Salt overwrites that file, Emacs will ask me again if I want to allow that theme.

In terms of Salt, maybe I need to use a finer-grained method than copying whole configuration files. Salt allows changing individual lines in config files, instead of overwriting the whole file. Copy the file if it doesn't exist; just update the relevant lines later?

I need to distill my dconf for GNOME programs installed on the system, as opposed to flatpaks, to be able to restore it later.

I'm sure there's detritus under ~/.config that should be managed by Salt... maybe I need to do another round of self-actualization and clean that up.

We have a couple of Windows machines kicking around at home, because schoolwork. Salt works on Windows, too, and I'd love to set them up for automatic backups to the home server.