Welcome to Richard's Blog

Author: applegater

0) Goals we achieved
- Main server runs WireGuard server (wg0)
- Pi1 + Pi2 run WireGuard clients
- Split tunnel (VPN traffic only; internet stays normal/fast)
- AdGuard Home runs on a Pi and keeps DNS control (no WG DNS override)
- Optional: Monitor Docker hosts from Uptime Kuma over WireGuard
- IPv6 permanently disabled (optional)
- UFW firewall locked down with only the ports you want public
1) WireGuard on the Main Server (Server side)

1.1 Install
```
sudo apt update
sudo apt install -y wireguard
```
1.2 Generate keys
```
wg genkey | sudo tee /etc/wireguard/server.key | wg pubkey | sudo tee /etc/wireguard/server.pub >/dev/null
sudo chmod 600 /etc/wireguard/server.key
```
1.3 Create /etc/wireguard/wg0.conf
```
sudo nano /etc/wireguard/wg0.conf
```
Paste:
```
[Interface]
Address = 10.8.0.1/24
ListenPort = 51820
PrivateKey = <SERVER_PRIVATE_KEY>
```
Insert the private key:
```
sudo cat /etc/wireguard/server.key
```
1.4 Enable + start
```
sudo systemctl enable wg-quick@wg0
sudo systemctl start wg-quick@wg0
sudo wg
```
2) WireGuard on Raspberry Pi 1 and Pi 2 (Client side)

Do this on each Pi, changing the IP.

2.1 Install
```
sudo apt update
sudo apt install -y wireguard
```
2.2 Generate keys
```
wg genkey | tee ~/client.key | wg pubkey > ~/client.pub
chmod 600 ~/client.key
```
2.3 Create /etc/wireguard/wg0.conf

Pi1 config (10.8.0.2)
```
sudo nano /etc/wireguard/wg0.conf
```
```
[Interface]
Address = 10.8.0.2/24
PrivateKey = <PI1_PRIVATE_KEY>
# IMPORTANT: No DNS line (because you run AdGuard)

[Peer]
PublicKey = <SERVER_PUBLIC_KEY>
Endpoint = <SERVER_PUBLIC_IP>:51820
AllowedIPs = 10.8.0.0/24
PersistentKeepalive = 25
```
Pi2 config (10.8.0.3)

Same, but:
```
Address = 10.8.0.3/24
PrivateKey = <PI2_PRIVATE_KEY>
```
✅ Critical split tunnel rule:
```
AllowedIPs = 10.8.0.0/24
```
❌ Do NOT use .../0 or 0.0.0.0/0.

2.4 Start on each Pi
```
sudo systemctl enable wg-quick@wg0
sudo systemctl start wg-quick@wg0
sudo wg
```
3) Add Pi peers to the Main Server

On the main server, open:
```
sudo nano /etc/wireguard/wg0.conf
```
Add:
```
[Peer]
PublicKey = <PI1_PUBLIC_KEY>
AllowedIPs = 10.8.0.2/32

[Peer]
PublicKey = <PI2_PUBLIC_KEY>
AllowedIPs = 10.8.0.3/32
```
Restart:
```
sudo systemctl restart wg-quick@wg0
sudo wg
```
4) Verify split tunnel is correct

4.1 Tunnel ping tests

From Pi1/Pi2:
```
ping 10.8.0.1
```
From server:
```
ping 10.8.0.2
ping 10.8.0.3
```
4.2 Confirm internet stays normal (split tunnel)

On Pi2:
```
ip route
```
You should see:
- default route via your LAN gateway (ex: 192.168.x.1)
- 10.8.0.0/24 dev wg0
If you ever see default via wg0, you accidentally full-tunneled.

5) DNS note (AdGuard Home on Pi)

You fixed this correctly.

✅ If AdGuard runs on the Pi, do NOT set DNS = ... in the WG config.
Because WireGuard would override system DNS and break resolution.

6) Optional: Disable IPv6 permanently (Ubuntu / Pi)

6.1 Sysctl method (recommended)
```
sudo nano /etc/sysctl.d/99-disable-ipv6.conf
```
Paste:
```
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
```
Apply:
```
sudo sysctl --system
```
Verify:
```
ip a | grep inet6
```
7) Firewall (UFW) with your current public ports (22, 80, 443)

7.1 Reset + defaults
```
sudo ufw disable
sudo ufw reset
sudo ufw default deny incoming
sudo ufw default allow outgoing
```
7.2 Allow required ports

SSH (choose one)

Safer (WireGuard only):
```
sudo ufw allow from 10.8.0.0/24 to any port 22 proto tcp comment "SSH via WireGuard"
```
Or if public SSH is needed:
```
sudo ufw limit 22/tcp comment "SSH rate limit"
```
HTTP/HTTPS:
```
sudo ufw allow 80/tcp comment "HTTP"
sudo ufw allow 443/tcp comment "HTTPS"
```
WireGuard port:
```
sudo ufw allow 51820/udp comment "WireGuard"
sudo ufw allow in on wg0 comment "WireGuard tunnel"
```
7.3 Enable
```
sudo ufw enable
sudo ufw status numbered
sudo ufw status verbose
```
8) Optional: Docker monitoring over WireGuard (Uptime Kuma)

8.1 Install docker socket proxy on each Pi (recommended)

Create docker-compose.yml:
```
version: "3.8"
services:
  docker-socket-proxy:
    image: tecnativa/docker-socket-proxy
    container_name: docker-socket-proxy
    ports:
      - "2375:2375"
    environment:
      CONTAINERS: 1
      INFO: 1
      PING: 1
      VERSION: 1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped
```
Run:
```
docker compose up -d
```
8.2 Firewall: allow Docker proxy ONLY over WireGuard

On each Pi:
```
sudo ufw allow from 10.8.0.0/24 to any port 2375 comment "Docker API via WireGuard only"
```
8.3 Add in Uptime Kuma
- Monitor Type: Docker Host
- URL:
  - http://10.8.0.2:2375
  - http://10.8.0.3:2375
9) What NOT to do (the pitfalls we hit)
- ❌ Don’t expose Docker 2375 publicly
- ❌ Don’t use AllowedIPs = .../0 (that becomes full tunnel)
- ❌ Don’t set DNS = 10.8.0.1 unless server truly runs DNS
- ❌ Don’t rely on Certbot/Caddy to “secure Docker API” (wrong model)
- ✅ Use WireGuard + firewall instead
Quick “health checklist” commands

On any node:
```
sudo wg
ip route
sudo ufw status verbose
ss -tulpen | head -n 30
```
If you want, I can also generate a one-page printable version of this runbook with:
- blanks for keys/IPs
- checkboxes
- and “expected outputs” for each command.
February 6, 2026
Secure Split-Tunnel WireGuard + AdGuard + UFW
This SOP is the final, stable operating procedure for a small production homelab using WireGuard, AdGuard Home, Docker monitoring, and UFW. It reflects all fixes, decisions, and lessons learned.

1) Design Goals
- Secure private management traffic
- No performance impact on internet traffic
- Public DNS via AdGuard Home (intentional)
- No public exposure of admin or Docker APIs
- Simple, predictable routing and firewall rules
2) Final Network Model
- Main Server: WireGuard server
- Raspberry Pi 1 / 2: WireGuard clients
- VPN Subnet: 10.8.0.0/24
- Tunnel Mode: Split tunnel (VPN traffic only)
- DNS: Local (AdGuard Home on Pi)
- IPv6: Disabled
- Firewall: UFW (IPv4 only)
3) WireGuard Configuration (FINAL)

Server (/etc/wireguard/wg0.conf)
```
[Interface]
Address = 10.8.0.1/24
ListenPort = 51820
PrivateKey = PLEASE_PUT_YOUR_SERVER_PRIVATE_KEY

[Peer]
PublicKey = PLEASE_PUT_YOUR_PI1_PUBLIC_KEY
AllowedIPs = 10.8.0.2/32

[Peer]
PublicKey = PLEASE_PUT_YOUR_PI2_PUBLIC_KEY
AllowedIPs = 10.8.0.3/32
```
Clients (Pi1 / Pi2)
```
[Interface]
Address = PLEASE_PUT_YOUR_PI_WG_IP
PrivateKey = PLEASE_PUT_YOUR_PI_PRIVATE_KEY
# No DNS line (AdGuard runs locally)

[Peer]
PublicKey = PLEASE_PUT_YOUR_SERVER_PUBLIC_KEY
Endpoint = PLEASE_PUT_YOUR_SERVER_PUBLIC_IP_OR_DNS:51820
AllowedIPs = 10.8.0.0/24
PersistentKeepalive = 25
```
✅ Critical rule: AllowedIPs = 10.8.0.0/24

❌ Never use /0 unless intentionally building a full-tunnel VPN.

4) Routing Verification (MANDATORY)

On each client:
```
ip route
```
Expected:
- Default route → LAN gateway
- 10.8.0.0/24 → wg0
If default route points to wg0, stop and fix before continuing.

5) DNS Model (FINAL)
- AdGuard Home runs locally on a Raspberry Pi
- WireGuard must not override DNS
- No DNS= entry in any WireGuard config
Public DNS services:
- 53/udp
- 53/tcp
- 853/tcp (DNS-over-TLS)
WireGuard DNS access is also allowed for internal clients.

6) IPv6 Policy (FINAL)

IPv6 is permanently disabled to reduce complexity and avoid dual-stack edge cases.
```
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
```
UFW IPv6 handling is disabled:
```
IPV6=no
```
7) Firewall Policy (UFW — FINAL)

Default Policy
```
ufw default deny incoming
ufw default allow outgoing
```
Publicly Exposed Ports (INTENTIONAL)
```
ufw allow 22/tcp        # SSH (rate-limited or WG-only)
ufw allow 80/tcp        # HTTP
ufw allow 443/tcp       # HTTPS
ufw allow 51820/udp     # WireGuard
ufw allow 53/udp        # DNS
ufw allow 53/tcp        # DNS
ufw allow 853/tcp       # DNS-over-TLS
```
WireGuard Internal Traffic
```
ufw allow in on wg0
ufw allow in on wg0 to any port 53
ufw allow in on wg0 to any port 853
```
SSH (Choose One)

WireGuard-only (recommended):
```
ufw allow from 10.8.0.0/24 to any port 22 proto tcp
```
OR public but rate-limited:
```
ufw limit 22/tcp
```
8) Docker Monitoring Policy (FINAL)
- Docker API is never public
- Access only over WireGuard
- Prefer docker-socket-proxy
Firewall rule:
```
ufw allow from 10.8.0.0/24 to any port 2375
```
Used by Uptime Kuma for monitoring only.

9) Automation & Defense-in-Depth
- AdGuard logs feed n8n automation
- Attacking IPs are auto-blocked
- Firewall provides first-layer filtering
- Automation provides adaptive response
Always whitelist:
- 10.8.0.0/24
- Admin IPs
- Health-check sources
10) Validation Checklist (RUN AFTER CHANGES)
```
wg
ip route
ping 10.8.0.1
ping 8.8.8.8
ping google.com
ufw status verbose
ss -tulpen | head -n 30
```
All must pass.

11) Known Pitfalls (DO NOT REPEAT)
- ❌ AllowedIPs = 0.0.0.0/0 (unintended full tunnel)
- ❌ Setting WireGuard DNS when AdGuard is local
- ❌ Exposing Docker ports publicly
- ❌ Using HTTPS proxies to “secure” Docker API
- ❌ Relying on UFW NAT side-effects for routing
12) Final State
- Fast internet (no tunnel overhead)
- Encrypted management traffic
- Public DNS intentionally exposed
- Minimal, auditable firewall rules
- No hidden routing or NAT dependencies
This SOP represents the final, correct configuration.
February 5, 2026
N8N Project for My production Server
Production Monitoring & Security Automation Runbook

Purpose

This runbook describes how to operate, monitor, and respond to events generated by the company’s production automation stack:
- n8n (orchestration)
- AdGuard DNS (Primary & Secondary Raspberry Pi)
- Fail2Ban
- Uptime Kuma
- Slack (alerting)
- Omada Controller (network devices)
It is written so that any on-call engineer can safely respond to alerts without deep system knowledge.

System Overview

What this system does
- Monitors DNS behavior on two AdGuard servers (Pi1 = Primary, Pi2 = Secondary)
- Detects possible DNS abuse / attacks using query heuristics
- Automatically blocks malicious IPs in AdGuard (when enabled)
- Monitors uptime of both DNS servers
- Pushes health heartbeats to Uptime Kuma
- Receives Fail2Ban ban/unban events from multiple hosts
- Receives Omada controller events (AP, gateway, switch up/down)
- Sends actionable alerts to Slack
What it does NOT do
- It does not permanently blacklist IPs without review
- It does not modify firewall rules (DNS-layer only)
- It does not auto-restart servers
Normal Operation (Healthy State)

Expected behavior
- Cron runs every minute
- Slack is quiet most of the time
- Uptime Kuma shows:
  - Pi1 Uptime: UP
  - Pi2 Uptime: UP
  - DNS Status: NORMAL
Normal Slack messages
- ✅ DNS NORMAL (baseline)
- ✅ DNS OK / RECOVERED
- ✅ Fail2Ban UNBANNED
- ℹ️ Omada informational events
No action is required in these cases.

Alert Types & Response Actions

🚨 POSSIBLE DNS ATTACK

Meaning
- One client is responsible for an abnormally high percentage of DNS queries
- Triggered when:
  - ≥ 80% of recent queries OR
  - ≥ 400 queries in sample window
Automatic actions
- AdGuard auto-block may already be applied
- IP reputation (IPinfo) is attached to the alert
Required response (step-by-step)
1. Open the Slack alert
2. Review:
  - Attacker IP
  - Client name (if known)
  - Organization / ASN
3. Log into the affected AdGuard server
4. Open Query Log
5. Confirm traffic pattern matches alert
6. If legitimate client:
  - Remove IP from disallowed_clients
  - Add client to DNS whitelist in n8n
7. If malicious:
  - No action needed (auto-block handled it)
Escalation
- Repeated attacks from different IPs → notify network/security team
✅ DNS OK / RECOVERED

Meaning
- DNS traffic has returned to normal
Action
- None required
🔴 / 🚨 UPTIME DOWN

Meaning
- DNS server is unreachable or returning bad HTTP status
Response steps
1. Check Uptime Kuma for confirmation
2. Attempt to reach host:
  - Ping
  - HTTPS access
3. If unreachable:
  - Check power
  - Check network connectivity
4. Review system logs if accessible
5. Restart service/server if required
Escalation
- If downtime > SLA threshold, notify management
🚫 Fail2Ban BANNED

Meaning
- Fail2Ban blocked an IP due to repeated authentication failures
Automatic actions
- IP already blocked at service level
- Geo/IP data added automatically
Response steps
1. Review IP reputation in Slack
2. Confirm jail name (sshd, nginx, etc.)
3. If internal or known IP:
  - Manually unban
  - Adjust Fail2Ban rules if needed
4. If external/malicious:
  - No action required
🚨 Omada Device DOWN

Meaning
- AP, gateway, or switch disconnected
Response steps
1. Identify device and site in Slack alert
2. Check Omada Controller status
3. Verify power and uplink
4. If multiple devices affected:
  - Suspect upstream outage
Environment & Configuration

Required environment variables (n8n)
- F2B_TOKEN
- IPINFO_TOKEN
- KUMA_PI1_UPTIME_URL
- KUMA_PI1_DNS_URL
- KUMA_PI2_UPTIME_URL
- KUMA_PI2_DNS_URL
Webhook endpoints
- /fail2ban-pi1
- /fail2ban-pi2
- /OmadaController
- /JoeOmadaTPlink
Maintenance & Safe Changes

Before making changes
- Disable auto-block if testing
- Clone workflow for testing
- Verify Slack output formatting
After changes
- Manually trigger workflow
- Confirm:
  - No duplicate Slack alerts
  - Kuma heartbeats still flow
Break-Glass (Emergency)

If automation behaves incorrectly:
1. Disable the n8n workflow
2. Remove IPs from AdGuard block list
3. Notify security/network team
4. Document incident
Ownership
- System owner: IT / Network Team
- Primary contact: IT Manager
- Slack channel: Monitoring / Security Alerts
February 4, 2026
How to secure our SSH with fail2ban 1GB Ram on Pi,
1. Update the system (important on Pi)
```
sudo apt update && sudo apt upgrade -y
```
Optional but recommended:
```
sudo reboot
```
2. Install Fail2Ban (very lightweight)
```
sudo apt install fail2ban -y
```
Memory usage on Pi: ~20–30 MB RAM idle ✅

3. Verify Fail2Ban is running
```
sudo systemctl status fail2ban
```
You should see:
```
Active: active (running)
```
If not:
```
sudo systemctl enable --now fail2ban
```
4. Create your local config (DO NOT edit defaults)

Fail2Ban updates will overwrite defaults — always use .local.
```
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
```
Or lighter (recommended for Pi):
```
sudo nano /etc/fail2ban/jail.local
```
Paste this minimal + safe config:
```
[DEFAULT]
bantime  = 1h
findtime = 10m
maxretry = 5
backend  = systemd
ignoreip = 127.0.0.1/8 ::1

[sshd]
enabled = true
port    = ssh
logpath = %(sshd_log)s
```
💡 backend = systemd is faster and more reliable on modern Pi OS.

5. Restart Fail2Ban
```
sudo systemctl restart fail2ban
```
6. Confirm SSH jail is active
```
sudo fail2ban-client status
```
Expected:
```
Jail list: sshd
```
Check banned IPs:
```
sudo fail2ban-client status sshd
```
7. Reduce resource usage (IMPORTANT for 1 GB Pi)

Edit:
```
sudo nano /etc/fail2ban/fail2ban.conf
```
Set:
```
loglevel = INFO
logtarget = /var/log/fail2ban.log
```
Optional: disable polling (systemd handles logs efficiently).

8. Enable log rotation (prevents SD wear)

Check:
```
sudo nano /etc/logrotate.d/fail2ban
```
Recommended content:
```
/var/log/fail2ban.log {
    weekly
    rotate 4
    compress
    missingok
    notifempty
}
```
9. Test Fail2Ban (safe test)

From another machine:
```
ssh wronguser@pi-ip
```
Fail 5 times → IP gets banned.

Unban yourself:
```
sudo fail2ban-client set sshd unbanip YOUR_IP
```
10. Optional hardening (strongly recommended)

Disable SSH passwords
```
sudo nano /etc/ssh/sshd_config
```
Set:
```
PasswordAuthentication no
PermitRootLogin no
```
Restart SSH:
```
sudo systemctl restart ssh
```
Summary (Pi-optimized)

✔ Low RAM usage
✔ SD-card friendly logging
✔ SSH protected
✔ systemd backend
✔ Safe upgrade-proof config
February 4, 2026