I'm a lousy sysadmin.
For years, my strategy for managing my machines in my home network has been one of maximum neglect:
-
Avoid distribution upgrades or reinstalls as much as possible, because stuff breaks or loses its configuration.
-
Keep my $HOME intact across upgrades, to preserve my personal configuration files as much as possible, even if it accumulates vast amounts of cruft.
-
Cry when I install a long-running service on my home server, like SMB shares or a music server, because I know it will break when I have to reinstall or upgrade.
About two years ago, I wrote some scripts to automate at least the package installation step, and to set particularly critical configuration files like firewall rules. These scripts helped me lose part of my fear of updates/reinstalls; after running them, I only needed to do a little manual work to get things back in working order. The scripts also made me start to think actively on what I am comfortable with in terms of distro defaults, and what I really need to change after a default installation.
Salt
In my mind there exists this whole universe of scary power tools for large-scale sysadmin work. Of course you would want some automation if you have a server farm to manage. Of course nobody would do this by hand if you had 3000 workstations somewhere. But for my puny home network with one server and two computers? Surely those tools are overkill?
Thankfully that is not so!
My colleague Richard Brown has been talking about the Salt Project for a few years. It is similar to tools for provisioning and configuration management like Ansible or Puppet.
What I have liked about Salt so far is that the documentation is very good, and it has let me translate my little setup into its configuration language while learning some good practices along the way.
I started with the Salt walkthrough, which is pretty nice.
TL;DR: the salt-master is the central box that keeps and distributes configuration to other machines, and those machines are called salt-minions. You write some mostly-declarative YAML in the salt-master, and propagate that configuration to the minions. Salt knows how to "create a user" or "install a package" or "restart a service when a config file changes" without you having to use distro-specific commands.
My home setup and how I want it to be
pambazo
- Has a RAID and serves media and stores backups. This is
my home server.
tlacoyo
- A desktop box, my main workstation.
torta
- My laptop, which has seen very little use during the
pandemic; in principle it should have an identical setup to my desktop
box.
I open the MDNS firewall ports on those three machines so I can use
somehost.local
to access them directly, without having to set up
DNS. Maybe I should learn how to do the latter.
All the machines need my basic configuration files (Emacs, shell prompt), and a few must-have programs (Emacs, Midnight Commander, git, podman).
My workstations need my gitlab/github SSH keys, my Suse VPN keys, some basic infrastructure for development, I need to be able to reinstall and reconstruct them quickly.
All the machines should get the same configuration for the two printers we have at home (a laser that can do two-sided printing, and an inkjet for photos).
My home server of course needs all the configuration for the various services it runs.
I also have a few short-lived virtual machines to test distro images. In those I only need my must-have packages.
Setting up the salt master
My home server works as the "salt master", which is the machine that
holds the configuration that should be distributed to other boxes. I
am just using the stock salt-master
package from openSUSE. The only
things I changed in its default configuration were the paths where it
looks for configuration, so I can use a git checkout in my home
directory instead of /etc/salt
. This goes in /etc/salt/master
:
# configuration for minions
file_roots:
base:
- /home/federico/src/salt-states
# sensitive data to be distributed to minions
pillar_roots:
base:
- /home/federico/src/salt-pillar
Setting up minions
It is easy enough to install the salt-minion
package and set up its
configuration to talk to the salt-master, but I wanted a way to
bootstrap that. Salt-bootstrap is exactly that. I can run this on a newly-installed machine:
curl -o bootstrap-salt.sh -L https://bootstrap.saltproject.io
sudo sh bootstrap-salt.sh -w -A 192.168.1.10 -i my-hostname stable
The first line downloads the bootstrap-salt.sh
script.
The second line:
-
-w
- use the distro's packages forsalt-minion
, not the upstream ones. -
-A 192.168.1.10
- the IP address of my salt-master. At this point the machine that is to become a minion doesn't have MDNS ports open yet, so it can't findpambazo.local
directly and it needs its IP address. -
-i my-hostname
- Name to give to the minion. Salt lets you have the minion's name different from the hostname, but I want them to be the same. That is, I want mytlacoyo.local
machine to be a minion calledtlacoyo
, etc. -
stable
- Use a stable release of salt-minion, not a development one.
When the script runs, it creates a keypair and asks the salt-master to register its public key. Then, on the salt-master I run this:
salt-key -a my-hostname
This accepts the minion's public key, and it is ready to be configured.
The very basics: set the hostname, open up MDNS in the firewall
I want my hostname to be the same as the minion name. Make it so!
'set hostname to be the same as the minion name':
network.system:
- hostname: {{ grains['id'] }}
- apply_hostname: True
- retain_settings: True
Salt uses Jinja templates to preprocess its configuration
files. One of the variables it makes available is grains
, which
contains information inherent to each minion: its CPU architecture,
amount of memory, OS distribution, and its minion id. Here I am using
{{ grains['id'] }}
to look up the minion id, and then set it as the
hostname.
To set up the firewall for desktop machines, I used YaST and then copied the resulting configuration to Salt:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
|
file.managed
is how Salt lets you copy files to destination
machines. In lines 1 to 6, salt://opensuse/desktop-firewalld.conf
gets copied to /etc/firewalld/firewalld.conf
. The salt://
prefix
indicates a path under your location for salt-states; this is the git
checkout with Salt's configuration that I mentioned above.
Lines 15 to 20 tell Salt to enable the firewalld
service, and to
restart it when either of two files change.
Indispensable packages
I cannot live without these:
'indispensable packages':
pkg.installed:
- pkgs:
- emacs
- git
- mc
- ripgrep
pkg.installed
takes an array of package names. Here I hard-code the
names which those packages have in openSUSE. Salt lets you do all
sorts of magic with Jinja and the salt-pillar mechanism to have
distro-specific package names, if you have a heterogeneous
environment. All my machines are openSUSE, so I don't need to do
that.
My username, and personal configuration files
This creates my user:
federico:
user.present:
- fullname: Federico Mena Quintero
- home: /home/federico
- shell: /bin/bash
- usergroup: False
- groups:
- users
This just copies a few configuration files:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
|
I use a Jinja array to define the list of files and their
destinations, and a for
loop to reduce the amount of typing.
Install some flatpaks
Install the flatpaks I need:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
|
Set up one of their configuration files:
/home/federico/.var/app/org.freac.freac/config/.freac/freac.xml:
file.managed:
- source: salt://opensuse/federico-config-files/freac.xml
- user: federico
- group: users
- mode: 644
- makedirs: True
This last file is the configuration for fre:ac, a CD audio
ripper. Flatpaks store their configuration files under
~/.var/app/flatpak-name
. I configured the app once by hand in its
GUI and then copied its configuration file to my salt-states.
Etcetera
The above is not all my setup; obviously things like the home server have a bunch of extra packages and configuration files. However, the patterns above are practically all I need to set up everything else.
How it works in practice
I have a git checkout of my salt-states. When I change them and I want to distribute the new configuration to my machines, I push to the git repository on my home server, and then just run a script that does this:
#!/bin/sh
set -e
cd /home/federico/src/salt-states
git pull
sudo salt '*' state.apply
The salt '*' state.apply
causes all machines to get an updated
configuration. Salt tells you what changed on each and what stayed
the same. The slowest part seems to be updating zypper
's package
repositories; apart from that, Salt is fast enough for me.
Playing with this for the first time
After setting up the bare salt-master on my home server, I created a virtual machine and immediately registered it as a salt-minion and created a snapshot for it. I wanted to go back to that "just installed, nothing set up" state easily to test my salt-states as if for a new setup. Once I was confident that it worked, I set up salt-minions on my desktop and laptop in exactly the same way. This was very useful!
A process of self-actualization
I have been gradually moving my accumulated configuration cruft from
my historical $HOME to Salt. This made me realize that I still had
obsolete dotfiles lying around like ~/.red-carpet
and
~/.realplayerrc
. That software is long gone. My dotfiles are much
cleaner now!
I was also able to remove my unused scripts in ~/bin
(I still had
the one I used to connect to Ximian's SSH tunnel, and the one I used
to connect to my university's PPP modem pool), realize which ones I
really need, move them to ~/.local/bin
and make them managed under
Salt.
To clean up that cruft across all machines, I have something like this:
/home/federico/.dotfile-that-is-no-longer-used:
file.absent
That deletes the file.
In the end I did reach my original goal: I can reinstall a machine, and then get it to a working state with a single command.
This has made me more confident to install cool toys in my home server... like a music server, which brings me endless joy. Look at all this!
One thing that would be nice in Flatpak
Salt lets you know what changed when you apply a configuration, or it can do a dry run where it tells you what will change without actually modifying anything. For example, Salt knows how to query the package database for each distro and tell you if a package needs to be updated, or if an existing config file is different from the one that will be propagated.
It would be nice if the flatpak
command-line tool would return this
information. In the examples above, I use flatpak
remote-add --if-not-exists
and flatpak install --or-update
to
achieve idempotency, but Salt is not able to know what Flatpak
actually did; it just runs the commands and returns success or failure.
Some pending things
Some programs keep state along with configuration in the same user-controlled files, and that state gets lost if each run of Salt just overwrites those files:
-
fre:ac, a CD audio ripper, has one of those "tip of the day" windows at startup. On each run, it updates the "number of the last shown tip" somewhere in its configuration for the user. However, since that configuration is in the same file that holds the paths for ripped music and the encoder parameters I want to use, the "tip of the day" gets reset to the beginning every time that Salt rewrites the config file.
-
When you load a theme in Emacs, say, with
custom-enabled-theme
incustom.el
, it stores a checksum of the theme you picked after first confirming that you indeed want to load the theme's code — to prevent malicious themes or something. However, that checksum is stored in another variable incustom.el
, so if Salt overwrites that file, Emacs will ask me again if I want to allow that theme.
In terms of Salt, maybe I need to use a finer-grained method than copying whole configuration files. Salt allows changing individual lines in config files, instead of overwriting the whole file. Copy the file if it doesn't exist; just update the relevant lines later?
I need to distill my dconf for GNOME programs installed on the system, as opposed to flatpaks, to be able to restore it later.
I'm sure there's detritus under ~/.config
that should be managed by
Salt... maybe I need to do another round of self-actualization and
clean that up.
We have a couple of Windows machines kicking around at home, because schoolwork. Salt works on Windows, too, and I'd love to set them up for automatic backups to the home server.