Welcome to Richard's Blog

Blog

Secure Split-Tunnel WireGuard + AdGuard + UFW
This SOP is the final, stable operating procedure for a small production homelab using WireGuard, AdGuard Home, Docker monitoring, and UFW. It reflects all fixes, decisions, and lessons learned.

1) Design Goals
- Secure private management traffic
- No performance impact on internet traffic
- Public DNS via AdGuard Home (intentional)
- No public exposure of admin or Docker APIs
- Simple, predictable routing and firewall rules
2) Final Network Model
- Main Server: WireGuard server
- Raspberry Pi 1 / 2: WireGuard clients
- VPN Subnet: 10.8.0.0/24
- Tunnel Mode: Split tunnel (VPN traffic only)
- DNS: Local (AdGuard Home on Pi)
- IPv6: Disabled
- Firewall: UFW (IPv4 only)
3) WireGuard Configuration (FINAL)

Server (/etc/wireguard/wg0.conf)
```
[Interface]
Address = 10.8.0.1/24
ListenPort = 51820
PrivateKey = PLEASE_PUT_YOUR_SERVER_PRIVATE_KEY

[Peer]
PublicKey = PLEASE_PUT_YOUR_PI1_PUBLIC_KEY
AllowedIPs = 10.8.0.2/32

[Peer]
PublicKey = PLEASE_PUT_YOUR_PI2_PUBLIC_KEY
AllowedIPs = 10.8.0.3/32
```
Clients (Pi1 / Pi2)
```
[Interface]
Address = PLEASE_PUT_YOUR_PI_WG_IP
PrivateKey = PLEASE_PUT_YOUR_PI_PRIVATE_KEY
# No DNS line (AdGuard runs locally)

[Peer]
PublicKey = PLEASE_PUT_YOUR_SERVER_PUBLIC_KEY
Endpoint = PLEASE_PUT_YOUR_SERVER_PUBLIC_IP_OR_DNS:51820
AllowedIPs = 10.8.0.0/24
PersistentKeepalive = 25
```
✅ Critical rule: AllowedIPs = 10.8.0.0/24

❌ Never use /0 unless intentionally building a full-tunnel VPN.

4) Routing Verification (MANDATORY)

On each client:
```
ip route
```
Expected:
- Default route → LAN gateway
- 10.8.0.0/24 → wg0
If default route points to wg0, stop and fix before continuing.

5) DNS Model (FINAL)
- AdGuard Home runs locally on a Raspberry Pi
- WireGuard must not override DNS
- No DNS= entry in any WireGuard config
Public DNS services:
- 53/udp
- 53/tcp
- 853/tcp (DNS-over-TLS)
WireGuard DNS access is also allowed for internal clients.

6) IPv6 Policy (FINAL)

IPv6 is permanently disabled to reduce complexity and avoid dual-stack edge cases.
```
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
```
UFW IPv6 handling is disabled:
```
IPV6=no
```
7) Firewall Policy (UFW — FINAL)

Default Policy
```
ufw default deny incoming
ufw default allow outgoing
```
Publicly Exposed Ports (INTENTIONAL)
```
ufw allow 22/tcp        # SSH (rate-limited or WG-only)
ufw allow 80/tcp        # HTTP
ufw allow 443/tcp       # HTTPS
ufw allow 51820/udp     # WireGuard
ufw allow 53/udp        # DNS
ufw allow 53/tcp        # DNS
ufw allow 853/tcp       # DNS-over-TLS
```
WireGuard Internal Traffic
```
ufw allow in on wg0
ufw allow in on wg0 to any port 53
ufw allow in on wg0 to any port 853
```
SSH (Choose One)

WireGuard-only (recommended):
```
ufw allow from 10.8.0.0/24 to any port 22 proto tcp
```
OR public but rate-limited:
```
ufw limit 22/tcp
```
8) Docker Monitoring Policy (FINAL)
- Docker API is never public
- Access only over WireGuard
- Prefer docker-socket-proxy
Firewall rule:
```
ufw allow from 10.8.0.0/24 to any port 2375
```
Used by Uptime Kuma for monitoring only.

9) Automation & Defense-in-Depth
- AdGuard logs feed n8n automation
- Attacking IPs are auto-blocked
- Firewall provides first-layer filtering
- Automation provides adaptive response
Always whitelist:
- 10.8.0.0/24
- Admin IPs
- Health-check sources
10) Validation Checklist (RUN AFTER CHANGES)
```
wg
ip route
ping 10.8.0.1
ping 8.8.8.8
ping google.com
ufw status verbose
ss -tulpen | head -n 30
```
All must pass.

11) Known Pitfalls (DO NOT REPEAT)
- ❌ AllowedIPs = 0.0.0.0/0 (unintended full tunnel)
- ❌ Setting WireGuard DNS when AdGuard is local
- ❌ Exposing Docker ports publicly
- ❌ Using HTTPS proxies to “secure” Docker API
- ❌ Relying on UFW NAT side-effects for routing
12) Final State
- Fast internet (no tunnel overhead)
- Encrypted management traffic
- Public DNS intentionally exposed
- Minimal, auditable firewall rules
- No hidden routing or NAT dependencies
This SOP represents the final, correct configuration.
February 5, 2026
N8N Project for My production Server
Production Monitoring & Security Automation Runbook

Purpose

This runbook describes how to operate, monitor, and respond to events generated by the company’s production automation stack:
- n8n (orchestration)
- AdGuard DNS (Primary & Secondary Raspberry Pi)
- Fail2Ban
- Uptime Kuma
- Slack (alerting)
- Omada Controller (network devices)
It is written so that any on-call engineer can safely respond to alerts without deep system knowledge.

System Overview

What this system does
- Monitors DNS behavior on two AdGuard servers (Pi1 = Primary, Pi2 = Secondary)
- Detects possible DNS abuse / attacks using query heuristics
- Automatically blocks malicious IPs in AdGuard (when enabled)
- Monitors uptime of both DNS servers
- Pushes health heartbeats to Uptime Kuma
- Receives Fail2Ban ban/unban events from multiple hosts
- Receives Omada controller events (AP, gateway, switch up/down)
- Sends actionable alerts to Slack
What it does NOT do
- It does not permanently blacklist IPs without review
- It does not modify firewall rules (DNS-layer only)
- It does not auto-restart servers
Normal Operation (Healthy State)

Expected behavior
- Cron runs every minute
- Slack is quiet most of the time
- Uptime Kuma shows:
  - Pi1 Uptime: UP
  - Pi2 Uptime: UP
  - DNS Status: NORMAL
Normal Slack messages
- ✅ DNS NORMAL (baseline)
- ✅ DNS OK / RECOVERED
- ✅ Fail2Ban UNBANNED
- ℹ️ Omada informational events
No action is required in these cases.

Alert Types & Response Actions

🚨 POSSIBLE DNS ATTACK

Meaning
- One client is responsible for an abnormally high percentage of DNS queries
- Triggered when:
  - ≥ 80% of recent queries OR
  - ≥ 400 queries in sample window
Automatic actions
- AdGuard auto-block may already be applied
- IP reputation (IPinfo) is attached to the alert
Required response (step-by-step)
1. Open the Slack alert
2. Review:
  - Attacker IP
  - Client name (if known)
  - Organization / ASN
3. Log into the affected AdGuard server
4. Open Query Log
5. Confirm traffic pattern matches alert
6. If legitimate client:
  - Remove IP from disallowed_clients
  - Add client to DNS whitelist in n8n
7. If malicious:
  - No action needed (auto-block handled it)
Escalation
- Repeated attacks from different IPs → notify network/security team
✅ DNS OK / RECOVERED

Meaning
- DNS traffic has returned to normal
Action
- None required
🔴 / 🚨 UPTIME DOWN

Meaning
- DNS server is unreachable or returning bad HTTP status
Response steps
1. Check Uptime Kuma for confirmation
2. Attempt to reach host:
  - Ping
  - HTTPS access
3. If unreachable:
  - Check power
  - Check network connectivity
4. Review system logs if accessible
5. Restart service/server if required
Escalation
- If downtime > SLA threshold, notify management
🚫 Fail2Ban BANNED

Meaning
- Fail2Ban blocked an IP due to repeated authentication failures
Automatic actions
- IP already blocked at service level
- Geo/IP data added automatically
Response steps
1. Review IP reputation in Slack
2. Confirm jail name (sshd, nginx, etc.)
3. If internal or known IP:
  - Manually unban
  - Adjust Fail2Ban rules if needed
4. If external/malicious:
  - No action required
🚨 Omada Device DOWN

Meaning
- AP, gateway, or switch disconnected
Response steps
1. Identify device and site in Slack alert
2. Check Omada Controller status
3. Verify power and uplink
4. If multiple devices affected:
  - Suspect upstream outage
Environment & Configuration

Required environment variables (n8n)
- F2B_TOKEN
- IPINFO_TOKEN
- KUMA_PI1_UPTIME_URL
- KUMA_PI1_DNS_URL
- KUMA_PI2_UPTIME_URL
- KUMA_PI2_DNS_URL
Webhook endpoints
- /fail2ban-pi1
- /fail2ban-pi2
- /OmadaController
- /JoeOmadaTPlink
Maintenance & Safe Changes

Before making changes
- Disable auto-block if testing
- Clone workflow for testing
- Verify Slack output formatting
After changes
- Manually trigger workflow
- Confirm:
  - No duplicate Slack alerts
  - Kuma heartbeats still flow
Break-Glass (Emergency)

If automation behaves incorrectly:
1. Disable the n8n workflow
2. Remove IPs from AdGuard block list
3. Notify security/network team
4. Document incident
Ownership
- System owner: IT / Network Team
- Primary contact: IT Manager
- Slack channel: Monitoring / Security Alerts
February 4, 2026
How to secure our SSH with fail2ban 1GB Ram on Pi,
1. Update the system (important on Pi)
```
sudo apt update && sudo apt upgrade -y
```
Optional but recommended:
```
sudo reboot
```
2. Install Fail2Ban (very lightweight)
```
sudo apt install fail2ban -y
```
Memory usage on Pi: ~20–30 MB RAM idle ✅

3. Verify Fail2Ban is running
```
sudo systemctl status fail2ban
```
You should see:
```
Active: active (running)
```
If not:
```
sudo systemctl enable --now fail2ban
```
4. Create your local config (DO NOT edit defaults)

Fail2Ban updates will overwrite defaults — always use .local.
```
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
```
Or lighter (recommended for Pi):
```
sudo nano /etc/fail2ban/jail.local
```
Paste this minimal + safe config:
```
[DEFAULT]
bantime  = 1h
findtime = 10m
maxretry = 5
backend  = systemd
ignoreip = 127.0.0.1/8 ::1

[sshd]
enabled = true
port    = ssh
logpath = %(sshd_log)s
```
💡 backend = systemd is faster and more reliable on modern Pi OS.

5. Restart Fail2Ban
```
sudo systemctl restart fail2ban
```
6. Confirm SSH jail is active
```
sudo fail2ban-client status
```
Expected:
```
Jail list: sshd
```
Check banned IPs:
```
sudo fail2ban-client status sshd
```
7. Reduce resource usage (IMPORTANT for 1 GB Pi)

Edit:
```
sudo nano /etc/fail2ban/fail2ban.conf
```
Set:
```
loglevel = INFO
logtarget = /var/log/fail2ban.log
```
Optional: disable polling (systemd handles logs efficiently).

8. Enable log rotation (prevents SD wear)

Check:
```
sudo nano /etc/logrotate.d/fail2ban
```
Recommended content:
```
/var/log/fail2ban.log {
    weekly
    rotate 4
    compress
    missingok
    notifempty
}
```
9. Test Fail2Ban (safe test)

From another machine:
```
ssh wronguser@pi-ip
```
Fail 5 times → IP gets banned.

Unban yourself:
```
sudo fail2ban-client set sshd unbanip YOUR_IP
```
10. Optional hardening (strongly recommended)

Disable SSH passwords
```
sudo nano /etc/ssh/sshd_config
```
Set:
```
PasswordAuthentication no
PermitRootLogin no
```
Restart SSH:
```
sudo systemctl restart ssh
```
Summary (Pi-optimized)

✔ Low RAM usage
✔ SD-card friendly logging
✔ SSH protected
✔ systemd backend
✔ Safe upgrade-proof config
February 4, 2026