greenbean.org

Andy Alexander — Linux Systems and Infrastructure

I'm a Linux systems professional with 20+ years of production experience. Since 2022 I've run that same discipline at home: designing, breaking, fixing, and documenting real infrastructure. Everything here is real and running — Proxmox across two sites, ZFS storage on TrueNAS SCALE, OPNsense edge routing over XGS-PON fiber, a WireGuard overlay network, redundant DNS, and automated verified backups.

What this demonstrates

Linux systems administration

systemd service management & troubleshooting
Package management, patching, OS upgrades
Filesystems, disks, users, groups, permissions
SSH, journald/log analysis, process & resource triage
Backup, restore, and service recovery

Infrastructure engineering

Virtualization — Proxmox, KVM/LXC
Networking, VLANs, firewalling, DNS/DHCP
Storage design — ZFS, RAID, NFS/SMB
Monitoring & observability
Automation — Ansible, Terraform, Python/Bash
Incident response & root-cause analysis

Selected work

Root-caused and resolved a full hypervisor hang on the primary Proxmox host, traced to a wedged USB controller shared with UPS monitoring
Diagnosed a same-host bridge networking bug that silently dropped one specific VM-to-VM TCP flow despite clean firewall/VLAN/FDB state at every layer checked; workaround routes the flow through the firewall instead (root cause still open)
Closed out an intermittent host scare on the primary Proxmox host with a full memory/MCE/RAS/EDAC investigation — no hardware fault confirmed, with a documented recurrence-testing plan
VLAN segmentation — flat 10.0.0.0/24 rebuilt into a segmented VLAN network (10.79.x.x) across the core switch, OPNsense, both Proxmox hosts, and the AP
30TB storage migration: mdadm RAID5 → TrueNAS SCALE / ZFS RAIDZ1 with PCIe SATA passthrough
WireGuard hub-and-spoke VPN — cloud-hosted hub, multi-site subnet routing, roaming clients
Automated restic backup pipeline across three Proxmox nodes, firewall infrastructure, and home automation — 90-day retention, scheduled integrity verification, off-site replication to Oracle Cloud
Replaced upstream DNS forwarding with per-host recursive, DNSSEC-validating Unbound resolvers on both Pi-holes
Migrated availability monitoring off broken unauthenticated Pi-hole checks to authenticated Push-based monitors after a Pi-hole v6 auth change

Current work

Prometheus + Grafana + node_exporter, alongside Uptime Kuma

Planned

Ansible fleet management — baseline, NUT, restic, and Docker host roles
Terraform coverage across the rest of the Proxmox fleet

Infrastructure at a glance

flowchart TB
    INET((Internet
XGS-PON Fiber))
    ONT[XGS-PON ONT-on-a-stick
SFP+ module]

    subgraph SW [MikroTik CRS310-8G+2S+ — RouterOS]
        BW[bridge-WAN
SFP+ cage + 1 port]
        BL[bridge-LAN
remaining ports]
    end

    subgraph WU [wu — Proxmox host, ODROID-H3]
        FW[OPNsense VM — Router / Firewall
Kea DHCPv4 · option 6 hands out both resolvers
WireGuard spoke]
        PH2[pihole2 · LXC 1000
Pi-hole, host install — secondary DNS]
    end

    PH1[pihole · ODROID-XU4
Armbian 26.8 / kernel 6.6
Pi-hole in Docker — primary DNS]

    subgraph SWG [swearengen — Proxmox host, i5-10600K / 48GB]
        NAS[TrueNAS SCALE VM
ZFS RAIDZ1 pool — PCIe SATA passthrough]
        FARN[farnum — Plex Media Server VM over NFS]
        HA[Home Assistant VM]
    end

    CLIENTS[LAN clients]

    KK1[kk1 · Oracle Cloud Ampere A1
WireGuard hub — static public endpoint
off-site restic target]

    subgraph REMOTE [Remote Site]
        ZOM[zombie · Debian VM
WireGuard spoke + subnet router]
        RPX[Remote Proxmox
Pi-hole + LinuxGSM game server]
    end

    INET --- ONT --- BW
    BW ---|"wu NIC 1 → WAN vNIC"| FW
    FW ---|"LAN vNIC → wu NIC 2"| BL
    BL --- CLIENTS
    BL --- PH1
    BL --- SWG
    CLIENTS -. "resolver 1" .-> PH1
    CLIENTS -. "resolver 2" .-> PH2
    FW == "WireGuard" ==> KK1
    ZOM == "WireGuard" ==> KK1
    ZOM --- RPX
    SWG -. restic over SFTP .- KK1

Documentation

homelab-docs — 23 runbooks, ~7,900 lines — each records what was built, why that approach, what broke, and how to rebuild it from nothing