boot: kernel-install owns /boot/loader; add lava host + test suite #40
Loading…
Reference in a new issue
No description provided.
Delete branch "william/lava"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Adds lava as a fleet host and moves
/boot/loaderunder systemdkernel-install, plus the test suite that makes fleet-wide boot changesreviewable without root on nine machines.
Boot policy: one declared mode per host
kernel/cmdline.tmplfails closed — no policy in host data means no/etc/kernel/cmdline, and an absent cmdline is nothing to boot with. Everyhost declares exactly one:
/etc/kernel/cmdlinegpt_auto = truerw4F68BCE3-…+LoaderDevicePartUUID;fstab.tmplrenders comment-onlykernel_install = true+kernel_cmdline = "…"/proc/cmdline, serial console kept)kernel_install = true, nokernel_cmdlinePENDING-AUDIT)Never
rootfstype=with a bcachefs root: gpt-auto-generator validates thefstype against
dissect_fstype_ok()'s allowlist, which excludes bcachefs →refused mount. Verified against systemd v261.3 source.
wak-kernel-install(audit | sync | adopt)Converges the ESP, driven by
pacman.d/hooks/99-wak-kernel-install.hookonkernel transactions and
run_onchange_wak-kernel-install.sh.tmplwhen bootconfig changes — the same two-trigger pattern as mkinitcpio. It pins
entry-token to machine-id, writes
loader/entries.srel = type1(without itkernel-installseeslayout=otherand writes nothing, andbootctl updatenever writes it), installs
/etc/kernel/loader.conf→/boot/loader/loader.conf,bootctl update,kernel-install addperinstalled kernel, and prunes only its own stale entries.
Two properties worth calling out, both source-verified in systemd v261.3:
bootctl updatenever overwrites an existingloader.conf(
bootctl-install.c:913), so the fleet owns that file once written.version=field sort below ours (boot.c:1798-1818) and stay aconsole-selectable fallback.
Why enabling
kernel_installfleet-wide in one commit is safeExisting hosts boot from hand-written entries whose options were never
recorded in this repo. Convergence is therefore blocked per host until a
human runs
/etc/wak-libexec/wak-kernel-install adoptat the console, whichrecords what the machine actually boots — not what the repo claims. If
kernel_cmdlinethen disagrees,syncrefuses loudly (rc 1, visible insidethe pacman transaction) until the data is fixed. Pending hosts keep booting
exactly what they boot today; nothing is left half-migrated. Onboarding steps
are in
docs/adding-a-host.md§5.igor carries real
kernel_cmdlinetoday; the other seven legacy hosts areintentionally in the pending state until each has been audited.
Deliberately not done
mkinitcpio.d/linux.presetis left alone. WithPRESETS=(), Arch's/usr/share/libalpm/scripts/mkinitcpioevaluates[[ -v PRESETS ]]asfalse (bash treats an empty array as unset) and falls through to its
hardcoded branch, which
rm -fs/boot/vmlinuz-linux+/boot/initramfs-linux.img— the live fallback pair. Keeping the presetcosts one duplicated initramfs build per kernel and keeps that fallback.
Tests (
test/, all run by the pre-commit hook)check-boot.py— new in the last commit: drives the migration statemachine against a fake ESP with stub
bootctl/kernel-install(29 checks:not-enabled, pending-options, pending-adoption, stale-adoption, converge,
idempotency, stale prune, legacy kept, and that
auditcatches an optionthe repo would silently drop). It found three real bugs while being
written.
render-machines.py/check-pristine.py— render igor/opus/lava in aprivate user+UTS+mount namespace (chezmoi has no hostname override, so
/etc/hostnameis bind-mounted from a fixture) and diff against committedsnapshots. Symlinks stored as text, ≥64 KiB files as sha256 blobs.
check-addresses.py—sec_addrunique fleet-wide,addrunique persite,
locknown,sec_addr⇄[host.X.sec], gateway ⇒addr. Loadshost data through chezmoi because Go-toml accepts multi-line inline
tables that strict
tomllibrejects..forgejo/workflows/test.yml— same suite onruns-on: arch.Notes for review
(including the
--security-opt seccomp=unconfinedrequirement for nestedunshare --userin the archlinux container, which I verified empirically)are in
test/README.md. Branch protection requiring these checks is a UIaction — specific-repository API tokens can't touch
/branch_protections/*./etc/wak-libexecis not onPATHanywhere in the fleet, so the docs usethe absolute path (as
prom-dmidecode.servicedoes).loc = "remote", wifi-primary: nolan_mac(eth is DHCP/SLAAC),addr = 4,sec_addr = 9, noendpoint(NATed — peers initiate). Noclevis/tang/TPM; the disk passphrase is typed at the physical console.
committed here): phases 06/08 updated to match this boot model.
test/render_lib.py renders each machine's full target state with the system chezmoi inside unshare --user --map-root-user --uts --mount, pinning hostname + /etc/{hostname,hosts,os-release} fixtures so .chezmoi.hostname resolves to the target machine on any Arch box — no root, real /etc untouched, --exclude scripts skips run_onchange_*. - current/<m> (gitignored) = fresh render; pristine/<m> = committed snapshots; files >= 64 KiB (rosenpass pqpk pubkeys) stored as content-addressed blobs/ with <name>.BLOB hash sidecars so key rotation diffs as one line; MANIFEST captures modes/symlinks/dirs (symlinks stored as text: /etc-style links are dangling elsewhere). - test/check-pristine.py: unified diff per machine, exit 1 on drift; machines listed but without a snapshot report SKIP (in-flight hosts). - test/update-pristine.py: refresh snapshots + prune unreferenced blobs. - test/hooks/pre-commit (core.hooksPath=test/hooks): renders from a git checkout-index of the staged tree and refuses commits on drift. Verified: clean run OK/OK/OK; injected nftables change produces 1-line drift per machine and is caught; harness caught its own .chezmoiignore regressions (docs//test/ leaking as managed dirs).systemd-boot entries were hand-written per host and the options they boot with were never recorded in this repo. Move /boot/loader under systemd kernel-install, driven by a fleet sync script on the same two-trigger pattern as mkinitcpio (pacman hook on kernel transactions + chezmoi run_onchange when boot config changes). Per-host policy, exactly one required (kernel/cmdline.tmpl fails closed otherwise — an absent /etc/kernel/cmdline means nothing to boot with): gpt_auto = true -> cmdline "rw"; root from the GPT root type GUID + LoaderDevicePartUUID kernel_install + kernel_cmdline -> those options verbatim (igor: the string verified from live /proc/cmdline, serial console kept) kernel_install, no cmdline -> comments-only PENDING-AUDIT marker; sync refuses to act wak-kernel-install (audit|sync|adopt) converges the ESP: entry-token pinned to machine-id, loader/entries.srel = type1 (without it kernel-install sees layout=other and writes nothing; bootctl update never writes it), /etc/kernel/loader.conf installed (bootctl update never overwrites an existing loader.conf — bootctl-install.c:913 — so we own it once written), bootctl update, kernel-install add per installed kernel, prune our own stale entries. Legacy manual entries are never touched: entries without version= sort below ours (boot.c:1798-1818) and stay the console fallback. Enabling kernel_install on all nine hosts in one commit is safe because convergence is blocked per host until `wak-kernel-install adopt` records what the machine actually boots; if the repo's kernel_cmdline then disagrees, sync refuses loudly (rc 1, visible in the pacman transaction). Pending hosts keep booting exactly what they do now. mkinitcpio.d/linux.preset is deliberately left alone: with PRESETS=(), Arch's alpm script sees [[ -v PRESETS ]] false (bash treats an empty array as unset) and falls to its hardcoded branch, which rm -fs the live /boot/vmlinuz-linux + initramfs-linux.img fallback pair. test/check-boot.py drives the state machine against a fake ESP with stub bootctl/kernel-install (29 checks: not-enabled, pending-options, pending-adoption, stale-adoption, converge, idempotency, stale prune, legacy kept, and that audit catches an option the repo would drop). Wired into the pre-commit hook and the Forgejo workflow. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>@ -1,5 +1,7 @@LICENSEREADME.mdAGENTS.mdFixed in
182ed38(rebased): that commit now removes only the barekernelignore line. TheAGENTS.mdanddocslines moved to3fa3f29, the commit that adds those paths. (Ignore comment 277 on this thread — malformed post from me, see the summary comment.)@ -1,5 +1,7 @@LICENSEREADME.mdAGENTS.mdBelongs in the commit adding AGENTS.md
@ -1,5 +1,7 @@LICENSEREADME.mdAGENTS.mddocsFixed in the rebase — the
docsignore line now lives in3fa3f29(docs: AGENTS.md + docs/adding-a-host.md), alongsideAGENTS.md.@ -1,5 +1,7 @@LICENSEREADME.mdAGENTS.mddocsBelongs in the commit adding docs
@ -0,0 +1,10 @@{{- $hd := index .host .chezmoi.hostname -}}{{- if and $hd (hasKey $hd "gpt_auto") $hd.gpt_auto -}}# Kernel command line for systemd-boot BLS entries (consumed byigor is pinned already: 2d179a8 carries
kernel_cmdline = "root=UUID=66486016-0742-4e12-b0d8-3bf79fb6236a systemd.gpt_auto=no console=tty0 console=ttyS1,115200n8", and the rendered result is snapshotted attest/machines/pristine/igor/kernel/cmdline.I could not capture the console specs for the other seven hosts from this checkout: no SSH credentials as
williamon them (delphi/exodus/elite/atomic/opus/level ->Permission denied (publickey), genius ->No route to host), and guessing options for a box I cannot read is exactly how a remote host gets stranded. Each needs its real cmdline:Give me working access, or paste the outputs, and I will land them as one follow-on commit adding
kernel_cmdlineper host. Until then those hosts render the comments-only PENDING-AUDIT file, sowak-kernel-installrefuses to touch their loader entries — pending is the safe state, not a broken one.@ -0,0 +1,10 @@{{- $hd := index .host .chezmoi.hostname -}}{{- if and $hd (hasKey $hd "gpt_auto") $hd.gpt_auto -}}# Kernel command line for systemd-boot BLS entries (consumed byOur current command line is like
Make sure in a follow on commit you add the console specifications and we add these to the machine definitions. This one is igor
@ -0,0 +2,4 @@loc = "remote"addr = 4sec_addr = 9# New-host boot policy: comment-only fstab + /etc/kernel/cmdline of `rw`;Done — feature prose is out of the data files: 60 comment lines removed across atomic/delphi/elite/exodus/genius/igor/level/opus in 2d179a8, and lava.toml cleaned in
cfd3bb8. The onboarding walkthrough stays in docs/adding-a-host.md §5, written once instead of duplicated nine times.Rule added to AGENTS.md in
3fa3f29:@ -0,0 +2,4 @@loc = "remote"addr = 4sec_addr = 9# New-host boot policy: comment-only fstab + /etc/kernel/cmdline of `rw`;Do not add these comments in the data files. Only brief information pertinent to the host itself. Add this rule to your agents md.
@ -0,0 +8,4 @@# install time and a systemd initrd with a bcachefs unlock hook.# See docs/adding-a-host.md §5.gpt_auto = true# no lan_mac: wifi-primary (iwd + networkd); ethernet is DHCP/SLAACRemoved. The
no lan_mac/no ipmi/no gateway/no endpointcomments are gone from lava.toml (cfd3bb8), and the3fa3f29rule covers it fleet-wide so they do not come back.@ -0,0 +8,4 @@# install time and a systemd initrd with a bcachefs unlock hook.# See docs/adding-a-host.md §5.gpt_auto = true# no lan_mac: wifi-primary (iwd + networkd); ethernet is DHCP/SLAACNo need to comment about what is missing, this might drift with time.
@ -0,0 +7,4 @@whatever `chezmoi source-path` reports under that config, typically`~/.local/share/chezetc-wak`).Hosts: igor, delphi, genius (mtv-w work/lab), exodus, atomic (gateways),Removed in
3fa3f29— the host roster and the forge/default-branch line are gone. AGENTS.md now just says to land everything through PRs against the default branch; hosts are discoverable from.chezmoidata/host/, the forge from the git remote.@ -0,0 +7,4 @@whatever `chezmoi source-path` reports under that config, typically`~/.local/share/chezetc-wak`).Hosts: igor, delphi, genius (mtv-w work/lab), exodus, atomic (gateways),Don't list the hosts or repo info, those can just be discovered from the repo itself.
@ -0,0 +32,4 @@## Template data model- Per-host data: `.chezmoidata/host/<name>.toml` with a `[host.<name>]`table (`loc`, `addr`, optional `sec_addr`, `lan_mac`, `nb_addr`,Done in
3fa3f29— the bullet no longer enumerates keys; it points at an existing host file for the shape and states the comment rule.@ -0,0 +32,4 @@## Template data model- Per-host data: `.chezmoidata/host/<name>.toml` with a `[host.<name>]`table (`loc`, `addr`, optional `sec_addr`, `lan_mac`, `nb_addr`,Don't list the parameters, just the filename reference is enough.
316156a2e953581e3cdcUpdate: my stray comments (277-283, 292-294) have been deleted. They were un-deletable only because I used the wrong path; the real one is
which returns 204. Editing a review comment genuinely is not offered (no PATCH on that path). Apologies for the noise.
All review feedback is landed on
53581e3; the only open item is capturing kernel_cmdline for the seven hosts I cannot SSH into from this checkout.Cleanup note: comments 292, 293 and 294 are mine, left while establishing how review replies thread on this instance (answer: same
path+new_position;in_reply_tois ignored). 294 is a correct threaded reply under "Belongs in the commit adding docs"; 292/293 were mechanism probes. As with 277-283, PATCH/DELETE on review comments returns 405 here, so I cannot remove them — deleting by hand would be appreciated.Everything actionable from the review is landed on
53581e3; the only open item is capturingkernel_cmdlinefor the seven hosts I cannot SSH into from this checkout.@ -1,5 +1,6 @@[host.atomic]loc = "abe-p"kernel_install = truedrop kernel_install from everywhere, we always want to do a kernel install. I will rework the hosts when I upgrade.
@ -1,5 +1,7 @@[host.igor]loc = "mtv-w"kernel_install = truekernel_cmdline = "root=UUID=66486016-0742-4e12-b0d8-3bf79fb6236a systemd.gpt_auto=no console=tty0 console=ttyS1,115200n8"Drop the root and systemd.gpt_auto, I will fix this on igor when I update
@ -0,0 +2,4 @@loc = "remote"addr = 4sec_addr = 9gpt_auto = trueDelete this everywhere, this is standard by default.
@ -3,0 +3,4 @@AGENTS.mddocstest.forgejoI don't think we need to ignore
.*files? chezmoi uses dot_ for dotfiles in /etc@ -0,0 +43,4 @@`{{- if and $hd (hasKey $hd "…") … -}}`. **Always `hasKey` before fieldaccess** — Go templates error on missing map keys (`index $hd "x"` is thenil-safe form; `$hd.x` on a missing key aborts the whole chezmoi run).- `.chezmoiignore.tmpl` is the gate for paths that must not exist on someIt should only be used for files that we don't understand how to manage, files that are conditionally exist on some hosts must be templates that are empty when not needed.
@ -0,0 +103,4 @@hostname + `/etc/{hostname,hosts,os-release}` fixtures — same safetycontract as the rootless render check, full-tree instead of single-template.## Boot policy — kernel-install owns /boot/loaderCan we split this into a separate doc. We need a reviewer and implementor guide for adding a host. You already have adding-a-host so maybe we just need a review?
@ -0,0 +17,4 @@| Question | Where it lands ||---|---|| **Which site** — `mtv-w` (work lab), `abe-p` (Apartment P), `remote` (NATed/offsite)? | `loc` → picks `dc.<loc>` prefixes; `remote` hosts get no `endpoint` (they dial out, peers with endpoints accept) and `remote`+gateway hosts get table-101 hairpin routes in `50-<net>.network.tmpl` |mtv-w (Mountain View William), abe-p (Allentown Parents), remote (no specific site location) -> change this to
roameverywhere.@ -0,0 +21,4 @@| **Ethernet or wifi-primary?** Wired hosts are on the managed VLAN LAN with a static lease keyed by MAC; wifi hosts join via iwd and get whatever the LAN's DHCP gives | wired: `lan_mac` (initrd static-IP + switch-side config); wifi: **no `lan_mac`** — every `50-*.network` template is `hasKey`-gated on it; WiFi config (iwd `main.conf` + host-side `60-*.network`) currently lives in the install runbook pre-chezmoi (lava: `~/lava/arch-install/05`), with a fleet iwd/`host/<name>/wifi/` template as the follow-up || **Gateway?** (site edge: NAT + wireguard gw net + routes) | `gateway = true` — only exodus (mtv-w) and atomic (abe-p) today; one per site || **Member of which overlay nets** — `sec` (management mesh, every fleet host), `gw` (site-mesh via gateway; every host that must reach other sites)? | `[host.<h>.sec]` / `[host.<h>.gw]` sections; a net section's presence *is* the membership flag everywhere in the templates || **Netbird?** (secondary mesh for console/VM access) | `nb_addr` (a `10.200.x.y` address allocated in the netbird admin network — allocate in netbird first, mirror here) |Make a commit to delete netbird earlier in the series, and remove it from all docs.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.