Fixes DNS resolution failures when persistent timers trigger backups
after system wake.
The NixOS borgbackup module adds network-online.target dependencies
to the timer when persistentTimer=true, but systemd timers don't pass
their dependencies to the services they trigger. This caused onsite
backups to start before the network was ready, resulting in "Could not
resolve hostname" errors.
Adding after/wants network-online.target directly to the service
ensures the backup waits for network availability regardless of how
it's triggered (timer or offsite's Wants= dependency).
Example failure (Oct 11, 07:43):
- Backup started at 07:43:43 (persistent timer caught up)
- DNS lookup failed: "Could not resolve hostname viridian.home.arpa"
- WiFi connected at 07:43:47 (4 seconds too late)
Applied to both fuchsia and viridian onsite backups.
Fixes multiple issues with borgbackup service coordination:
1. Race condition between onsite/offsite backups
- Set Type=oneshot to ensure services wait for completion
- Added Wants= dependency to trigger onsite when offsite runs
- Prevents snapshot path collision at /btrfs-subvolumes
2. Network unavailability after sleep/wake
- Added persistentTimer=true to onsite backups
- NixOS module now auto-adds network-online.target dependencies
- Fixes DNS resolution failures for SSH repos
3. Data loss risk from missed backups
- Persistent timers ensure backups run on wake if missed
- Protects work done before sleep from being unbackored
4. Duplicate onsite runs at midnight
- Removed 15-minute stagger (00:15 -> 00:00)
- Systemd deduplicates services in same transaction
- Onsite now runs once, not twice
Applied to both fuchsia and viridian for consistency.
Update backup paths to use actual persistent storage locations (/persist/*) rather than bind-mounted paths, making it clear where data truly resides and simplifying restore operations.
Changes staging directories from hidden to visible and aligns backup paths with actual BTRFS subvolume naming conventions for better clarity when browsing archives.
Adds persistentTimer=true to both fuchsia and viridian offsite backup configurations to ensure backups run on next boot if the system was asleep at the scheduled time. Without this, daily backups would be skipped entirely until the next scheduled run.
Fixes backup system authentication and hostname resolution issues.
Changes:
- Change PermitRootLogin from "no" to "prohibit-password" in global SSH config
(allows key-based root login for host-to-host backups while blocking passwords)
- Update fuchsia onsite backup to use viridian.home.arpa FQDN instead of shortname
- Update SSH knownHosts to use FQDNs (fuchsia.home.arpa, viridian.home.arpa)
(system-level config uses FQDNs, user shortcuts remain in home-manager)
This enables the complete 3-2-1 backup strategy with automated backups working
correctly between fuchsia and viridian, and fuchsia to BorgBase.
Restructures SSH trust relationships from global to host-specific configuration
for better locality of concern and principle of least privilege.
Changes:
- Collapse nixos/common/global/ssh/ back to ssh.nix (single-file module)
- Move internal host trust (fuchsia/viridian) to per-host services/ssh/
- Split BorgBase known hosts by repository (li9kg944 for fuchsia, r7ag7x1w for viridian)
- Add viridian SSH server config to accept backup connections from fuchsia
- Add fuchsia borgbackup passphrase for offsite backups
- Configure viridian to create /srv/borg-repo/fuchsia for remote backups
This enables the 3-2-1 backup strategy with fuchsia backing up to both viridian
(onsite) and BorgBase (offsite) with proper SSH authentication.
Add critical system state from persist.nix to borgbackup jobs:
- SSH host keys (required for borg authentication)
- machine-id and nixos state
- Network and bluetooth configurations
Paths mirror persist.nix configuration for maintainability.
Service-specific persist data (traefik, crowdsec) excluded -
will create dedicated subvolumes if/when needed.
Add automated snapshot and backup system with three independent tiers:
Snapper (hourly local snapshots):
- Configure snapper for all srv-* subvolumes
- Tiered retention: 24 hourly, 7 daily, 4 weekly, 12 monthly
- Snapshots stored at /.snapshots on viridian drive
- Provides fast operational rollback for user errors
Borgbackup onsite (hourly local backups):
- Independent staging snapshots at /.staging-onsite
- Repository on data drive at /srv/borg-repo
- Unencrypted (physical security assumed)
- Matches snapper retention policy
- Fast local disaster recovery
Borgbackup offsite (daily remote backups):
- Independent staging snapshots at /.staging-offsite
- Encrypted backups to borgbase repository
- Retention: 7 daily, 4 weekly, 12 monthly
- Remote disaster recovery with prune policy
Architecture decisions:
- Separate staging directories prevent job conflicts
- Staging snapshots decouple borg jobs from snapper
- Consistent zstd,9 compression across both borg jobs
- Special case handling for containers subvolume path
Migrate from path-based persistence (/persist/var/lib/*) to dedicated
BTRFS subvolumes for better data isolation and snapshot capabilities.
- Move valuable user-facing services to /srv/* with srv-* subvolumes:
- forgejo: git repositories and database
- opengist: paste data
- minecraft: game world data
- lighttpd: static web content
- containers: OCI container volumes
- Update home directory to use hm-sajenim subvolume on viridian disk
- Remove jupyterhub service (no longer in use)
- Update borgbackup paths to match new service locations
- Follow upstream service defaults where possible for maintainability
Services kept on /persist (disposable state):
- traefik, crowdsec, murmur
Move all allowUnfreePredicate declarations to global configs to prevent
the "last definition wins" merging issue. Unfree packages are now managed
in two central locations:
- NixOS system packages: nixos/common/global/default.nix
- Home Manager packages: home-manager/sajenim/global/default.nix
- Remove ollama service configuration and dependencies
- Clean up traefik routing for ollama web interface
- Comment out traefik service examples for clarity
