Commit graph

11 commits

Author SHA1 Message Date
d21b36a1b0
fix(borgbackup): remove persistent timers to prevent post-resume failures
Removes persistentTimer from all borgbackup services and unnecessary
network-online.target dependencies. Changes fuchsia offsite to 14:00
fixed schedule when system is reliably awake.

Persistent timer catch-ups immediately after system resume caused
failures due to services starting before network/system fully stabilized:
- Onsite: DNS resolution failures (viridian.home.arpa)
- Offsite: BorgBase connection refusals during SSH/borg handshake

Fixed schedules provide reliable backups without catch-up complexity:
- fuchsia offsite: 14:00 daily (typical awake time for desktop)
- viridian offsite: midnight daily (always-on server)
- All onsite: hourly (no catch-up needed)

Offsite services retain wants/after dependencies on onsite completion
to prevent race conditions on shared /btrfs-subvolumes snapshot paths.

Network dependencies removed as fixed schedules run when system is
already stable, eliminating timing issues with network-online.target.
2025-10-14 09:59:27 +08:00
a276fdf53a
fix(borgbackup): prevent race conditions and improve reliability
Fixes multiple issues with borgbackup service coordination:

1. Race condition between onsite/offsite backups
   - Set Type=oneshot to ensure services wait for completion
   - Added Wants= dependency to trigger onsite when offsite runs
   - Prevents snapshot path collision at /btrfs-subvolumes

2. Network unavailability after sleep/wake
   - Added persistentTimer=true to onsite backups
   - NixOS module now auto-adds network-online.target dependencies
   - Fixes DNS resolution failures for SSH repos

3. Data loss risk from missed backups
   - Persistent timers ensure backups run on wake if missed
   - Protects work done before sleep from being unbackored

4. Duplicate onsite runs at midnight
   - Removed 15-minute stagger (00:15 -> 00:00)
   - Systemd deduplicates services in same transaction
   - Onsite now runs once, not twice

Applied to both fuchsia and viridian for consistency.
2025-10-09 11:30:00 +08:00
15b4851e8e
refactor(borgbackup): implement shared staging with defense-in-depth
Major improvements to borgbackup configuration for better reliability and
maintainability:

**Shared staging directory:**
- Use single /btrfs-subvolumes directory (was /subvolumes-{onsite,offsite})
- Eliminates redundant path suffixes in archive structure
- Archive paths now semantic: /btrfs-subvolumes/srv-forgejo clearly indicates
  BTRFS subvolume content without redundant backup job metadata

**Defense-in-depth protection:**
- Layer 1: Systemd ordering - offsite waits for onsite completion
- Layer 2: Self-healing preHook - auto-cleanup orphaned snapshots from
  crashes/power loss
- Prevents cascading failures from race conditions or abnormal terminations

**Code quality improvements:**
- Extract subvolume lists to reduce duplication (DRY principle)
- Add /* sh */ syntax hints for proper editor highlighting
- Silent operation for consistency with existing hooks
- Improved readability with clearer comments and formatting
- All lines ≤ 100 characters

**Timing:**
- Offsite: *-*-* 00:15:00 (daily at 12:15 AM, waits for onsite)
- Onsite: hourly (unchanged)
2025-10-08 18:46:50 +08:00
37924375a2
refactor(borgbackup): backup from /persist paths instead of bind mounts
Update backup paths to use actual persistent storage locations (/persist/*) rather than bind-mounted paths, making it clear where data truly resides and simplifying restore operations.
2025-10-08 15:58:23 +08:00
26c08000a0
refactor(borgbackup): use visible directories with semantic subvolume names
Changes staging directories from hidden to visible and aligns backup paths with actual BTRFS subvolume naming conventions for better clarity when browsing archives.
2025-10-08 15:53:15 +08:00
359d01c407
fix(borgbackup): enable persistent timers for offsite backups
Adds persistentTimer=true to both fuchsia and viridian offsite backup configurations to ensure backups run on next boot if the system was asleep at the scheduled time. Without this, daily backups would be skipped entirely until the next scheduled run.
2025-10-08 08:04:57 +08:00
acab920858
WIP: SSH configuration restructure
Backup of SSH reorganization changes for future reference.
2025-10-07 20:58:09 +08:00
a6fa8866ac
feat(fuchsia): implement backup strategy with explicit home paths
Add snapper and borgbackup for fuchsia home directory backups:

Snapper Configuration:
- Hourly snapshots of /home/sajenim
- Retention: 24 hourly, 7 daily, 4 weekly, 12 monthly
- Stored in nested .snapshots subvolume

Borgbackup Onsite:
- Backup to viridian over SSH (local network)
- Target: ssh://viridian/srv/borg-repo/fuchsia
- Hourly backups, unencrypted, deduplicated
- Same retention as snapper

Borgbackup Offsite:
- Backup to borgbase (internet)
- Target: li9kg944@li9kg944.repo.borgbase.com:repo
- Daily backups, encrypted (repokey-blake2), deduplicated
- Retention: 7 daily, 4 weekly, 12 monthly

Explicit Home Paths (valuable user data only):
- Documents, Pictures, Videos, Music, Downloads, Academics, Notes
- Dotfiles: .ssh, .gnupg

System Persist Data:
- SSH host keys, machine-id, nixos state
- Bluetooth, NetworkManager configurations

Intentionally Excluded:
- .config (managed declaratively via home-manager)
- .repositories (cloneable from GitHub)
- .cache and build artifacts

Treats viridian as central backup server, maintaining 3-2-1 strategy
(3 copies, 2 locations, 1 offsite).

chore(viridian): remove unused inputs parameter from borgbackup offsite
2025-10-07 19:14:11 +08:00
f24a7476a7
feat(viridian): add explicit persist data to backup strategy
Add critical system state from persist.nix to borgbackup jobs:
- SSH host keys (required for borg authentication)
- machine-id and nixos state
- Network and bluetooth configurations

Paths mirror persist.nix configuration for maintainability.
Service-specific persist data (traefik, crowdsec) excluded -
will create dedicated subvolumes if/when needed.
2025-10-07 17:06:45 +08:00
7833d89d86
fix(viridian): resolve backup system initialization issues
Fix snapper and borgbackup jobs to work with ephemeral-btrfs setup:

Snapper fixes:
- Remove global /.snapshots mount (use nested subvolumes instead)
- Remove unused hostname variable
- Snapshots now stored in .snapshots subvolumes within each service

Borgbackup fixes:
- Add systemd.tmpfiles.rules to create staging directories at boot
- Add readWritePaths for staging directories (systemd sandboxing)
- Staging directories survive ephemeral root wipes

Architecture notes:
- Nested .snapshots subvolumes don't require separate mounts
- systemd tmpfiles ensures directories exist before services start
- ProtectSystem=strict requires explicit ReadWritePaths allowlist
2025-10-07 09:38:07 +08:00
c05598d9e0
feat(viridian): implement comprehensive 3-2-1 backup strategy
Add automated snapshot and backup system with three independent tiers:

Snapper (hourly local snapshots):
- Configure snapper for all srv-* subvolumes
- Tiered retention: 24 hourly, 7 daily, 4 weekly, 12 monthly
- Snapshots stored at /.snapshots on viridian drive
- Provides fast operational rollback for user errors

Borgbackup onsite (hourly local backups):
- Independent staging snapshots at /.staging-onsite
- Repository on data drive at /srv/borg-repo
- Unencrypted (physical security assumed)
- Matches snapper retention policy
- Fast local disaster recovery

Borgbackup offsite (daily remote backups):
- Independent staging snapshots at /.staging-offsite
- Encrypted backups to borgbase repository
- Retention: 7 daily, 4 weekly, 12 monthly
- Remote disaster recovery with prune policy

Architecture decisions:
- Separate staging directories prevent job conflicts
- Staging snapshots decouple borg jobs from snapper
- Consistent zstd,9 compression across both borg jobs
- Special case handling for containers subvolume path
2025-10-06 20:59:26 +08:00