greenbean.org

Andy Alexander — Linux Systems and Infrastructure

I'm a Linux systems professional with 20+ years of production experience. Since 2022 I've run that same discipline at home: designing, breaking, fixing, and documenting real infrastructure. Everything here is real and running — Proxmox at home and at a remote client site, ZFS storage on TrueNAS SCALE, OPNsense edge routing over XGS-PON fiber, a WireGuard overlay network, redundant DNS, and automated verified backups.

Live right now: tasks.greenbean.org — a Postgres-backed app on a 3-node k3s cluster, provisioned by Terraform + Ansible, monitored by Icinga2, and served over a self-managed Nginx/Let's Encrypt reverse proxy. Open it, add a task, watch the status panel in the corner.

What this demonstrates

Linux systems administration

• systemd service management & troubleshooting
• Package management, patching, OS upgrades
• Filesystems, disks, users, groups, permissions
• SSH, journald/log analysis, process & resource triage
• Backup, restore, and service recovery

Infrastructure engineering

• Virtualization — Proxmox, KVM/LXC, Kubernetes (k3s)
• Networking, VLANs, firewalling, DNS/DHCP
• Storage design — ZFS, RAID, NFS/SMB
• Monitoring & observability — Icinga2, Uptime Kuma
• Automation — Terraform, Ansible, Python/Bash
• Incident response & root-cause analysis

Selected work

• Provisioned a 3-node Kubernetes (k3s) cluster with supporting PostgreSQL and Icinga2 hosts entirely via Terraform and Ansible, deploying a containerized Flask/PostgreSQL application exposed publicly at tasks.greenbean.org over a self-managed Nginx/Let's Encrypt reverse proxy
• Root-caused and resolved a full hypervisor hang on the primary Proxmox host, traced to a wedged USB controller shared with UPS monitoring
• Diagnosed a same-host bridge networking bug that silently dropped one specific VM-to-VM TCP flow despite clean firewall/VLAN/FDB state at every layer checked; workaround routes the flow through the firewall instead (root cause still open)
• Closed out an intermittent host scare on the primary Proxmox host with a full memory/MCE/RAS/EDAC investigation — no hardware fault confirmed, with a documented recurrence-testing plan
• VLAN segmentation — flat 10.0.0.0/24 rebuilt into a segmented VLAN network (10.79.x.x) across the core switch, OPNsense, both Proxmox hosts, and the AP
• 30TB storage migration: mdadm RAID5 → TrueNAS SCALE / ZFS RAIDZ1 with PCIe SATA passthrough
• WireGuard hub-and-spoke VPN — cloud-hosted hub, multi-site subnet routing, roaming clients
• Automated restic backup pipeline across three Proxmox nodes, firewall infrastructure, and home automation — 90-day retention, scheduled integrity verification, off-site replication to Oracle Cloud
• Replaced upstream DNS forwarding with per-host recursive, DNSSEC-validating Unbound resolvers on both Pi-holes
• Migrated availability monitoring off broken unauthenticated Pi-hole checks to authenticated Push-based monitors after a Pi-hole v6 auth change

Current work

• Prometheus + Grafana + node_exporter, alongside Uptime Kuma

Planned

• Extend Ansible fleet roles — baseline, NUT, restic, and Docker host roles — to swearengen and wu
• Extend Terraform coverage to the rest of the Proxmox fleet

Infrastructure at a glance

flowchart TB
    INET((Internet
XGS-PON Fiber))
    ONT[XGS-PON ONT-on-a-stick
SFP+ module]

    subgraph SW [MikroTik CRS310-8G+2S+ — RouterOS]
        BW[bridge-WAN
SFP+ cage + 1 port]
        BL[bridge-LAN
remaining ports]
    end

    subgraph WU [wu — Proxmox host, ODROID-H3]
        FW[OPNsense VM — Router / Firewall
Kea DHCPv4 · option 6 hands out both resolvers
WireGuard spoke]
        PH2[pihole2 · LXC 1000
Pi-hole, host install — secondary DNS]
    end

    PH1[pihole · ODROID-XU4
Armbian 26.8 / kernel 6.6
Pi-hole in Docker — primary DNS]

    subgraph SWG [swearengen — Proxmox host, i5-10600K / 48GB]
        NAS[TrueNAS SCALE VM
ZFS RAIDZ1 pool — PCIe SATA passthrough]
        FARN[farnum — Plex Media Server VM over NFS]
        HA[Home Assistant VM]
    end

    CLIENTS[LAN clients]

    KK1[kk1 · Oracle Cloud Ampere A1
WireGuard hub — static public endpoint
off-site restic target]

    subgraph REMOTE [Remote client site]
        ZOM[zombie · Debian VM
WireGuard spoke + subnet router]
        RPX[Remote Proxmox
Pi-hole + LinuxGSM game server]
    end

    INET --- ONT --- BW
    BW ---|"wu NIC 1 → WAN vNIC"| FW
    FW ---|"LAN vNIC → wu NIC 2"| BL
    BL --- CLIENTS
    BL --- PH1
    BL --- SWG
    CLIENTS -. "resolver 1" .-> PH1
    CLIENTS -. "resolver 2" .-> PH2
    FW == "WireGuard" ==> KK1
    ZOM == "WireGuard" ==> KK1
    ZOM --- RPX
    SWG -. restic over SFTP .- KK1

Documentation

homelab-docs — 23 runbooks, ~7,900 lines — each records what was built, why that approach, what broke, and how to rebuild it from nothing