Category: Linux Server

  • Building a Secure, Split‑Tunnel WireGuard Homelab (End‑to‑End)

    This page documents the complete, final configuration of a secure homelab built with WireGuard, Raspberry Pis, UFW, AdGuard Home, Fail2Ban, n8n automation, and Uptime Kuma. It is written as a from‑top‑to‑bottom reference: design intent, implementation, validation, and final security posture.

    The goal is not maximum complexity, but clear, intentional security that is easy to operate and reason about.


    Design Goals

    • Secure all management access using WireGuard
    • Preserve full internet speed (no full‑tunnel VPN)
    • Expose only explicitly intended public services
    • Eliminate accidental routing, NAT, and DNS side effects
    • Provide monitoring, alerting, and automated response
    • Keep the system auditable and maintainable

    Final Architecture Overview

    • Main Server: WireGuard server and central management node
    • Raspberry Pi 1 & 2: WireGuard clients and service hosts
    • VPN Subnet: 10.8.0.0/24
    • Tunnel Mode: Split tunnel (VPN traffic only)
    • DNS: Local (AdGuard Home on Raspberry Pi)
    • Firewall: UFW (IPv4 only)
    • IPv6: Disabled intentionally

    Only traffic destined for the VPN subnet traverses WireGuard. All normal internet traffic continues to use the local gateway.


    WireGuard Configuration (Final)

    Server — /etc/wireguard/wg0.conf

    [Interface]
    Address = 10.8.0.1/24
    ListenPort = 51820
    PrivateKey = PLEASE_PUT_YOUR_SERVER_PRIVATE_KEY
    
    [Peer]
    PublicKey = PLEASE_PUT_YOUR_PI1_PUBLIC_KEY
    AllowedIPs = 10.8.0.2/32
    
    [Peer]
    PublicKey = PLEASE_PUT_YOUR_PI2_PUBLIC_KEY
    AllowedIPs = 10.8.0.3/32
    

    Raspberry Pi Clients — /etc/wireguard/wg0.conf

    [Interface]
    Address = PLEASE_PUT_YOUR_PI_WG_IP
    PrivateKey = PLEASE_PUT_YOUR_PI_PRIVATE_KEY
    # No DNS line (AdGuard Home runs locally)
    
    [Peer]
    PublicKey = PLEASE_PUT_YOUR_SERVER_PUBLIC_KEY
    Endpoint = PLEASE_PUT_YOUR_SERVER_PUBLIC_IP_OR_DNS:51820
    AllowedIPs = 10.8.0.0/24
    PersistentKeepalive = 25
    

    Critical rule: AllowedIPs = 10.8.0.0/24

    Using /0 would unintentionally create a full‑tunnel VPN and introduce NAT dependencies. The /24 mask ensures a true split tunnel.


    Routing Validation (Required)

    On each Raspberry Pi:

    ip route
    

    Expected output:

    • Default route → LAN gateway
    • 10.8.0.0/24wg0

    If the default route points to wg0, stop and correct the configuration before continuing.


    DNS Design (AdGuard Home)

    • AdGuard Home runs locally on a Raspberry Pi
    • WireGuard does not override DNS
    • No DNS= directive exists in WireGuard configs

    Public DNS services are intentionally exposed:

    • 53/udp, 53/tcp — DNS
    • 853/tcp — DNS‑over‑TLS

    DNS logs feed automation for abuse detection and response.


    IPv6 Policy

    IPv6 is intentionally disabled to reduce complexity and avoid dual‑stack routing and DNS edge cases common in small environments.

    Sysctl — /etc/sysctl.d/99-disable-ipv6.conf

    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv6.conf.default.disable_ipv6 = 1
    net.ipv6.conf.lo.disable_ipv6 = 1
    

    UFW IPv6 toggle — /etc/default/ufw

    IPV6=no
    

    The environment operates entirely over IPv4.


    Firewall Policy (UFW — Final)

    Default Policy

    ufw default deny incoming
    ufw default allow outgoing
    

    Publicly Exposed (Intentional)

    • 22/tcp — SSH (WireGuard‑only or rate‑limited)
    • 80/tcp — HTTP
    • 443/tcp — HTTPS
    • 51820/udp — WireGuard
    • 53/udp, 53/tcp — DNS
    • 853/tcp — DNS‑over‑TLS

    WireGuard Internal Access

    ufw allow in on wg0
    ufw allow in on wg0 to any port 53
    ufw allow in on wg0 to any port 853
    

    SSH Access Model

    Preferred (WireGuard‑only):

    ufw allow from 10.8.0.0/24 to any port 22 proto tcp
    

    External scans and LAN SSH attempts correctly show blocked.


    SSH Hardening

    • SSH key‑only authentication
    • Password authentication disabled
    • Root login avoided or disabled
    • SSH reachable only via WireGuard IPs

    Ping confirms network reachability. SSH access requires correct user and authorized key placement.


    Docker Monitoring (Uptime Kuma)

    • Docker API is never public
    • Accessed only over WireGuard
    • Exposed via read‑only docker‑socket‑proxy

    Firewall rule:

    ufw allow from 10.8.0.0/24 to any port 2375
    

    This allows monitoring without exposing control capabilities.


    Detection, Alerting, and Automation

    • Fail2Ban blocks brute‑force attempts
    • AdGuard Home logs capture DNS abuse
    • n8n processes events every minute
    • Automated IP blocking is applied
    • Alerts are delivered to Slack
    • Uptime Kuma monitors hosts and containers

    This provides a full detect → alert → respond pipeline.


    Validation Checklist

    wg
    ip route
    ping 10.8.0.1
    ping 8.8.8.8
    ping google.com
    ufw status verbose
    ss -tulpen | head -n 30
    

    External port scans for SSH should show blocked. SSH access works only from WireGuard peers using WireGuard IPs.


    Common Pitfalls (Avoided)

    • AllowedIPs = 0.0.0.0/0 (unintended full tunnel)
    • Overriding DNS when AdGuard runs locally
    • Exposing Docker APIs publicly
    • Using HTTPS proxies as Docker security
    • Relying on firewall NAT side effects

    Final Summary

    This setup results in a secure, split‑tunnel WireGuard network between a main server and multiple Raspberry Pis, while keeping performance high and avoiding unnecessary complexity.

    The main server acts as the WireGuard server, and each Raspberry Pi connects as a client on a private VPN subnet (10.8.0.0/24). Only internal VPN traffic is routed through WireGuard, while normal internet traffic continues to use each device’s local gateway. This design avoids speed degradation and removes the need for NAT or full‑tunnel routing.

    DNS handling is intentionally local. Because one Raspberry Pi runs AdGuard Home, WireGuard does not override system DNS settings, preventing common resolution issues and keeping behavior predictable.

    IPv6 is permanently disabled to reduce complexity and avoid dual‑stack edge cases. The firewall exposes only explicitly intended services, and SSH access is restricted to WireGuard peers using key‑only authentication.

    Monitoring, alerting, and automated response are handled through Uptime Kuma, Fail2Ban, and n8n, providing real‑time visibility and protection.

    The final result is a fast, secure, low‑maintenance homelab with a clearly defined attack surface, intentional access paths, and documented operating procedures — designed for reliability rather than complexity.

  • 0) Goals we achieved

    • Main server runs WireGuard server (wg0)
    • Pi1 + Pi2 run WireGuard clients
    • Split tunnel (VPN traffic only; internet stays normal/fast)
    • AdGuard Home runs on a Pi and keeps DNS control (no WG DNS override)
    • Optional: Monitor Docker hosts from Uptime Kuma over WireGuard
    • IPv6 permanently disabled (optional)
    • UFW firewall locked down with only the ports you want public

    1) WireGuard on the Main Server (Server side)

    1.1 Install

    sudo apt update
    sudo apt install -y wireguard
    

    1.2 Generate keys

    wg genkey | sudo tee /etc/wireguard/server.key | wg pubkey | sudo tee /etc/wireguard/server.pub >/dev/null
    sudo chmod 600 /etc/wireguard/server.key
    

    1.3 Create /etc/wireguard/wg0.conf

    sudo nano /etc/wireguard/wg0.conf
    

    Paste:

    [Interface]
    Address = 10.8.0.1/24
    ListenPort = 51820
    PrivateKey = <SERVER_PRIVATE_KEY>
    

    Insert the private key:

    sudo cat /etc/wireguard/server.key
    

    1.4 Enable + start

    sudo systemctl enable wg-quick@wg0
    sudo systemctl start wg-quick@wg0
    sudo wg
    

    2) WireGuard on Raspberry Pi 1 and Pi 2 (Client side)

    Do this on each Pi, changing the IP.

    2.1 Install

    sudo apt update
    sudo apt install -y wireguard
    

    2.2 Generate keys

    wg genkey | tee ~/client.key | wg pubkey > ~/client.pub
    chmod 600 ~/client.key
    

    2.3 Create /etc/wireguard/wg0.conf

    Pi1 config (10.8.0.2)

    sudo nano /etc/wireguard/wg0.conf
    
    [Interface]
    Address = 10.8.0.2/24
    PrivateKey = <PI1_PRIVATE_KEY>
    # IMPORTANT: No DNS line (because you run AdGuard)
    
    [Peer]
    PublicKey = <SERVER_PUBLIC_KEY>
    Endpoint = <SERVER_PUBLIC_IP>:51820
    AllowedIPs = 10.8.0.0/24
    PersistentKeepalive = 25
    

    Pi2 config (10.8.0.3)

    Same, but:

    Address = 10.8.0.3/24
    PrivateKey = <PI2_PRIVATE_KEY>
    

    Critical split tunnel rule:

    AllowedIPs = 10.8.0.0/24
    

    ❌ Do NOT use .../0 or 0.0.0.0/0.

    2.4 Start on each Pi

    sudo systemctl enable wg-quick@wg0
    sudo systemctl start wg-quick@wg0
    sudo wg
    

    3) Add Pi peers to the Main Server

    On the main server, open:

    sudo nano /etc/wireguard/wg0.conf
    

    Add:

    [Peer]
    PublicKey = <PI1_PUBLIC_KEY>
    AllowedIPs = 10.8.0.2/32
    
    [Peer]
    PublicKey = <PI2_PUBLIC_KEY>
    AllowedIPs = 10.8.0.3/32
    

    Restart:

    sudo systemctl restart wg-quick@wg0
    sudo wg
    

    4) Verify split tunnel is correct

    4.1 Tunnel ping tests

    From Pi1/Pi2:

    ping 10.8.0.1
    

    From server:

    ping 10.8.0.2
    ping 10.8.0.3
    

    4.2 Confirm internet stays normal (split tunnel)

    On Pi2:

    ip route
    

    You should see:

    • default route via your LAN gateway (ex: 192.168.x.1)
    • 10.8.0.0/24 dev wg0

    If you ever see default via wg0, you accidentally full-tunneled.


    5) DNS note (AdGuard Home on Pi)

    You fixed this correctly.

    ✅ If AdGuard runs on the Pi, do NOT set DNS = ... in the WG config.
    Because WireGuard would override system DNS and break resolution.


    6) Optional: Disable IPv6 permanently (Ubuntu / Pi)

    6.1 Sysctl method (recommended)

    sudo nano /etc/sysctl.d/99-disable-ipv6.conf
    

    Paste:

    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv6.conf.default.disable_ipv6 = 1
    net.ipv6.conf.lo.disable_ipv6 = 1
    

    Apply:

    sudo sysctl --system
    

    Verify:

    ip a | grep inet6
    

    7) Firewall (UFW) with your current public ports (22, 80, 443)

    7.1 Reset + defaults

    sudo ufw disable
    sudo ufw reset
    sudo ufw default deny incoming
    sudo ufw default allow outgoing
    

    7.2 Allow required ports

    SSH (choose one)

    Safer (WireGuard only):

    sudo ufw allow from 10.8.0.0/24 to any port 22 proto tcp comment "SSH via WireGuard"
    

    Or if public SSH is needed:

    sudo ufw limit 22/tcp comment "SSH rate limit"
    

    HTTP/HTTPS:

    sudo ufw allow 80/tcp comment "HTTP"
    sudo ufw allow 443/tcp comment "HTTPS"
    

    WireGuard port:

    sudo ufw allow 51820/udp comment "WireGuard"
    sudo ufw allow in on wg0 comment "WireGuard tunnel"
    

    7.3 Enable

    sudo ufw enable
    sudo ufw status numbered
    sudo ufw status verbose
    

    8) Optional: Docker monitoring over WireGuard (Uptime Kuma)

    8.1 Install docker socket proxy on each Pi (recommended)

    Create docker-compose.yml:

    version: "3.8"
    services:
      docker-socket-proxy:
        image: tecnativa/docker-socket-proxy
        container_name: docker-socket-proxy
        ports:
          - "2375:2375"
        environment:
          CONTAINERS: 1
          INFO: 1
          PING: 1
          VERSION: 1
        volumes:
          - /var/run/docker.sock:/var/run/docker.sock:ro
        restart: unless-stopped
    

    Run:

    docker compose up -d
    

    8.2 Firewall: allow Docker proxy ONLY over WireGuard

    On each Pi:

    sudo ufw allow from 10.8.0.0/24 to any port 2375 comment "Docker API via WireGuard only"
    

    8.3 Add in Uptime Kuma

    • Monitor Type: Docker Host
    • URL:
      • http://10.8.0.2:2375
      • http://10.8.0.3:2375

    9) What NOT to do (the pitfalls we hit)

    • ❌ Don’t expose Docker 2375 publicly
    • ❌ Don’t use AllowedIPs = .../0 (that becomes full tunnel)
    • ❌ Don’t set DNS = 10.8.0.1 unless server truly runs DNS
    • ❌ Don’t rely on Certbot/Caddy to “secure Docker API” (wrong model)
    • ✅ Use WireGuard + firewall instead

    Quick “health checklist” commands

    On any node:

    sudo wg
    ip route
    sudo ufw status verbose
    ss -tulpen | head -n 30
    

    If you want, I can also generate a one-page printable version of this runbook with:

    • blanks for keys/IPs
    • checkboxes
    • and “expected outputs” for each command.
  • Secure Split-Tunnel WireGuard + AdGuard + UFW

    This SOP is the final, stable operating procedure for a small production homelab using WireGuard, AdGuard Home, Docker monitoring, and UFW. It reflects all fixes, decisions, and lessons learned.


    1) Design Goals

    • Secure private management traffic
    • No performance impact on internet traffic
    • Public DNS via AdGuard Home (intentional)
    • No public exposure of admin or Docker APIs
    • Simple, predictable routing and firewall rules

    2) Final Network Model

    • Main Server: WireGuard server
    • Raspberry Pi 1 / 2: WireGuard clients
    • VPN Subnet: 10.8.0.0/24
    • Tunnel Mode: Split tunnel (VPN traffic only)
    • DNS: Local (AdGuard Home on Pi)
    • IPv6: Disabled
    • Firewall: UFW (IPv4 only)

    3) WireGuard Configuration (FINAL)

    Server (/etc/wireguard/wg0.conf)

    [Interface]
    Address = 10.8.0.1/24
    ListenPort = 51820
    PrivateKey = PLEASE_PUT_YOUR_SERVER_PRIVATE_KEY
    
    [Peer]
    PublicKey = PLEASE_PUT_YOUR_PI1_PUBLIC_KEY
    AllowedIPs = 10.8.0.2/32
    
    [Peer]
    PublicKey = PLEASE_PUT_YOUR_PI2_PUBLIC_KEY
    AllowedIPs = 10.8.0.3/32
    

    Clients (Pi1 / Pi2)

    [Interface]
    Address = PLEASE_PUT_YOUR_PI_WG_IP
    PrivateKey = PLEASE_PUT_YOUR_PI_PRIVATE_KEY
    # No DNS line (AdGuard runs locally)
    
    [Peer]
    PublicKey = PLEASE_PUT_YOUR_SERVER_PUBLIC_KEY
    Endpoint = PLEASE_PUT_YOUR_SERVER_PUBLIC_IP_OR_DNS:51820
    AllowedIPs = 10.8.0.0/24
    PersistentKeepalive = 25
    

    Critical rule: AllowedIPs = 10.8.0.0/24

    ❌ Never use /0 unless intentionally building a full-tunnel VPN.


    4) Routing Verification (MANDATORY)

    On each client:

    ip route
    

    Expected:

    • Default route → LAN gateway
    • 10.8.0.0/24wg0

    If default route points to wg0, stop and fix before continuing.


    5) DNS Model (FINAL)

    • AdGuard Home runs locally on a Raspberry Pi
    • WireGuard must not override DNS
    • No DNS= entry in any WireGuard config

    Public DNS services:

    • 53/udp
    • 53/tcp
    • 853/tcp (DNS-over-TLS)

    WireGuard DNS access is also allowed for internal clients.


    6) IPv6 Policy (FINAL)

    IPv6 is permanently disabled to reduce complexity and avoid dual-stack edge cases.

    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv6.conf.default.disable_ipv6 = 1
    net.ipv6.conf.lo.disable_ipv6 = 1
    

    UFW IPv6 handling is disabled:

    IPV6=no
    

    7) Firewall Policy (UFW — FINAL)

    Default Policy

    ufw default deny incoming
    ufw default allow outgoing
    

    Publicly Exposed Ports (INTENTIONAL)

    ufw allow 22/tcp        # SSH (rate-limited or WG-only)
    ufw allow 80/tcp        # HTTP
    ufw allow 443/tcp       # HTTPS
    ufw allow 51820/udp     # WireGuard
    ufw allow 53/udp        # DNS
    ufw allow 53/tcp        # DNS
    ufw allow 853/tcp       # DNS-over-TLS
    

    WireGuard Internal Traffic

    ufw allow in on wg0
    ufw allow in on wg0 to any port 53
    ufw allow in on wg0 to any port 853
    

    SSH (Choose One)

    WireGuard-only (recommended):

    ufw allow from 10.8.0.0/24 to any port 22 proto tcp
    

    OR public but rate-limited:

    ufw limit 22/tcp
    

    8) Docker Monitoring Policy (FINAL)

    • Docker API is never public
    • Access only over WireGuard
    • Prefer docker-socket-proxy

    Firewall rule:

    ufw allow from 10.8.0.0/24 to any port 2375
    

    Used by Uptime Kuma for monitoring only.


    9) Automation & Defense-in-Depth

    • AdGuard logs feed n8n automation
    • Attacking IPs are auto-blocked
    • Firewall provides first-layer filtering
    • Automation provides adaptive response

    Always whitelist:

    • 10.8.0.0/24
    • Admin IPs
    • Health-check sources

    10) Validation Checklist (RUN AFTER CHANGES)

    wg
    ip route
    ping 10.8.0.1
    ping 8.8.8.8
    ping google.com
    ufw status verbose
    ss -tulpen | head -n 30
    

    All must pass.


    11) Known Pitfalls (DO NOT REPEAT)

    • AllowedIPs = 0.0.0.0/0 (unintended full tunnel)
    • ❌ Setting WireGuard DNS when AdGuard is local
    • ❌ Exposing Docker ports publicly
    • ❌ Using HTTPS proxies to “secure” Docker API
    • ❌ Relying on UFW NAT side-effects for routing

    12) Final State

    • Fast internet (no tunnel overhead)
    • Encrypted management traffic
    • Public DNS intentionally exposed
    • Minimal, auditable firewall rules
    • No hidden routing or NAT dependencies

    This SOP represents the final, correct configuration.

  • N8N Project for My production Server

    Production Monitoring & Security Automation Runbook

    Purpose

    This runbook describes how to operate, monitor, and respond to events generated by the company’s production automation stack:

    • n8n (orchestration)
    • AdGuard DNS (Primary & Secondary Raspberry Pi)
    • Fail2Ban
    • Uptime Kuma
    • Slack (alerting)
    • Omada Controller (network devices)

    It is written so that any on-call engineer can safely respond to alerts without deep system knowledge.


    System Overview

    What this system does

    • Monitors DNS behavior on two AdGuard servers (Pi1 = Primary, Pi2 = Secondary)
    • Detects possible DNS abuse / attacks using query heuristics
    • Automatically blocks malicious IPs in AdGuard (when enabled)
    • Monitors uptime of both DNS servers
    • Pushes health heartbeats to Uptime Kuma
    • Receives Fail2Ban ban/unban events from multiple hosts
    • Receives Omada controller events (AP, gateway, switch up/down)
    • Sends actionable alerts to Slack

    What it does NOT do

    • It does not permanently blacklist IPs without review
    • It does not modify firewall rules (DNS-layer only)
    • It does not auto-restart servers

    Normal Operation (Healthy State)

    Expected behavior

    • Cron runs every minute
    • Slack is quiet most of the time
    • Uptime Kuma shows:
      • Pi1 Uptime: UP
      • Pi2 Uptime: UP
      • DNS Status: NORMAL

    Normal Slack messages

    • ✅ DNS NORMAL (baseline)
    • ✅ DNS OK / RECOVERED
    • ✅ Fail2Ban UNBANNED
    • ℹ️ Omada informational events

    No action is required in these cases.


    Alert Types & Response Actions

    🚨 POSSIBLE DNS ATTACK

    Meaning

    • One client is responsible for an abnormally high percentage of DNS queries
    • Triggered when:
      • ≥ 80% of recent queries OR
      • ≥ 400 queries in sample window

    Automatic actions

    • AdGuard auto-block may already be applied
    • IP reputation (IPinfo) is attached to the alert

    Required response (step-by-step)

    1. Open the Slack alert
    2. Review:
      • Attacker IP
      • Client name (if known)
      • Organization / ASN
    3. Log into the affected AdGuard server
    4. Open Query Log
    5. Confirm traffic pattern matches alert
    6. If legitimate client:
      • Remove IP from disallowed_clients
      • Add client to DNS whitelist in n8n
    7. If malicious:
      • No action needed (auto-block handled it)

    Escalation

    • Repeated attacks from different IPs → notify network/security team

    ✅ DNS OK / RECOVERED

    Meaning

    • DNS traffic has returned to normal

    Action

    • None required

    🔴 / 🚨 UPTIME DOWN

    Meaning

    • DNS server is unreachable or returning bad HTTP status

    Response steps

    1. Check Uptime Kuma for confirmation
    2. Attempt to reach host:
      • Ping
      • HTTPS access
    3. If unreachable:
      • Check power
      • Check network connectivity
    4. Review system logs if accessible
    5. Restart service/server if required

    Escalation

    • If downtime > SLA threshold, notify management

    🚫 Fail2Ban BANNED

    Meaning

    • Fail2Ban blocked an IP due to repeated authentication failures

    Automatic actions

    • IP already blocked at service level
    • Geo/IP data added automatically

    Response steps

    1. Review IP reputation in Slack
    2. Confirm jail name (sshd, nginx, etc.)
    3. If internal or known IP:
      • Manually unban
      • Adjust Fail2Ban rules if needed
    4. If external/malicious:
      • No action required

    🚨 Omada Device DOWN

    Meaning

    • AP, gateway, or switch disconnected

    Response steps

    1. Identify device and site in Slack alert
    2. Check Omada Controller status
    3. Verify power and uplink
    4. If multiple devices affected:
      • Suspect upstream outage

    Environment & Configuration

    Required environment variables (n8n)

    • F2B_TOKEN
    • IPINFO_TOKEN
    • KUMA_PI1_UPTIME_URL
    • KUMA_PI1_DNS_URL
    • KUMA_PI2_UPTIME_URL
    • KUMA_PI2_DNS_URL

    Webhook endpoints

    • /fail2ban-pi1
    • /fail2ban-pi2
    • /OmadaController
    • /JoeOmadaTPlink

    Maintenance & Safe Changes

    Before making changes

    • Disable auto-block if testing
    • Clone workflow for testing
    • Verify Slack output formatting

    After changes

    • Manually trigger workflow
    • Confirm:
      • No duplicate Slack alerts
      • Kuma heartbeats still flow

    Break-Glass (Emergency)

    If automation behaves incorrectly:

    1. Disable the n8n workflow
    2. Remove IPs from AdGuard block list
    3. Notify security/network team
    4. Document incident

    Ownership

    • System owner: IT / Network Team
    • Primary contact: IT Manager
    • Slack channel: Monitoring / Security Alerts

Secret Link