Blog

  • N8N Project for My production Server

    Production Monitoring & Security Automation Runbook

    Purpose

    This runbook describes how to operate, monitor, and respond to events generated by the company’s production automation stack:

    • n8n (orchestration)
    • AdGuard DNS (Primary & Secondary Raspberry Pi)
    • Fail2Ban
    • Uptime Kuma
    • Slack (alerting)
    • Omada Controller (network devices)

    It is written so that any on-call engineer can safely respond to alerts without deep system knowledge.


    System Overview

    What this system does

    • Monitors DNS behavior on two AdGuard servers (Pi1 = Primary, Pi2 = Secondary)
    • Detects possible DNS abuse / attacks using query heuristics
    • Automatically blocks malicious IPs in AdGuard (when enabled)
    • Monitors uptime of both DNS servers
    • Pushes health heartbeats to Uptime Kuma
    • Receives Fail2Ban ban/unban events from multiple hosts
    • Receives Omada controller events (AP, gateway, switch up/down)
    • Sends actionable alerts to Slack

    What it does NOT do

    • It does not permanently blacklist IPs without review
    • It does not modify firewall rules (DNS-layer only)
    • It does not auto-restart servers

    Normal Operation (Healthy State)

    Expected behavior

    • Cron runs every minute
    • Slack is quiet most of the time
    • Uptime Kuma shows:
      • Pi1 Uptime: UP
      • Pi2 Uptime: UP
      • DNS Status: NORMAL

    Normal Slack messages

    • ✅ DNS NORMAL (baseline)
    • ✅ DNS OK / RECOVERED
    • ✅ Fail2Ban UNBANNED
    • ℹ️ Omada informational events

    No action is required in these cases.


    Alert Types & Response Actions

    🚨 POSSIBLE DNS ATTACK

    Meaning

    • One client is responsible for an abnormally high percentage of DNS queries
    • Triggered when:
      • ≥ 80% of recent queries OR
      • ≥ 400 queries in sample window

    Automatic actions

    • AdGuard auto-block may already be applied
    • IP reputation (IPinfo) is attached to the alert

    Required response (step-by-step)

    1. Open the Slack alert
    2. Review:
      • Attacker IP
      • Client name (if known)
      • Organization / ASN
    3. Log into the affected AdGuard server
    4. Open Query Log
    5. Confirm traffic pattern matches alert
    6. If legitimate client:
      • Remove IP from disallowed_clients
      • Add client to DNS whitelist in n8n
    7. If malicious:
      • No action needed (auto-block handled it)

    Escalation

    • Repeated attacks from different IPs → notify network/security team

    ✅ DNS OK / RECOVERED

    Meaning

    • DNS traffic has returned to normal

    Action

    • None required

    🔴 / 🚨 UPTIME DOWN

    Meaning

    • DNS server is unreachable or returning bad HTTP status

    Response steps

    1. Check Uptime Kuma for confirmation
    2. Attempt to reach host:
      • Ping
      • HTTPS access
    3. If unreachable:
      • Check power
      • Check network connectivity
    4. Review system logs if accessible
    5. Restart service/server if required

    Escalation

    • If downtime > SLA threshold, notify management

    🚫 Fail2Ban BANNED

    Meaning

    • Fail2Ban blocked an IP due to repeated authentication failures

    Automatic actions

    • IP already blocked at service level
    • Geo/IP data added automatically

    Response steps

    1. Review IP reputation in Slack
    2. Confirm jail name (sshd, nginx, etc.)
    3. If internal or known IP:
      • Manually unban
      • Adjust Fail2Ban rules if needed
    4. If external/malicious:
      • No action required

    🚨 Omada Device DOWN

    Meaning

    • AP, gateway, or switch disconnected

    Response steps

    1. Identify device and site in Slack alert
    2. Check Omada Controller status
    3. Verify power and uplink
    4. If multiple devices affected:
      • Suspect upstream outage

    Environment & Configuration

    Required environment variables (n8n)

    • F2B_TOKEN
    • IPINFO_TOKEN
    • KUMA_PI1_UPTIME_URL
    • KUMA_PI1_DNS_URL
    • KUMA_PI2_UPTIME_URL
    • KUMA_PI2_DNS_URL

    Webhook endpoints

    • /fail2ban-pi1
    • /fail2ban-pi2
    • /OmadaController
    • /JoeOmadaTPlink

    Maintenance & Safe Changes

    Before making changes

    • Disable auto-block if testing
    • Clone workflow for testing
    • Verify Slack output formatting

    After changes

    • Manually trigger workflow
    • Confirm:
      • No duplicate Slack alerts
      • Kuma heartbeats still flow

    Break-Glass (Emergency)

    If automation behaves incorrectly:

    1. Disable the n8n workflow
    2. Remove IPs from AdGuard block list
    3. Notify security/network team
    4. Document incident

    Ownership

    • System owner: IT / Network Team
    • Primary contact: IT Manager
    • Slack channel: Monitoring / Security Alerts

  • How to secure our SSH with fail2ban 1GB Ram on Pi,

    1. Update the system (important on Pi)

    sudo apt update && sudo apt upgrade -y
    

    Optional but recommended:

    sudo reboot
    

    2. Install Fail2Ban (very lightweight)

    sudo apt install fail2ban -y
    

    Memory usage on Pi: ~20–30 MB RAM idle


    3. Verify Fail2Ban is running

    sudo systemctl status fail2ban
    

    You should see:

    Active: active (running)
    

    If not:

    sudo systemctl enable --now fail2ban
    

    4. Create your local config (DO NOT edit defaults)

    Fail2Ban updates will overwrite defaults — always use .local.

    sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
    

    Or lighter (recommended for Pi):

    sudo nano /etc/fail2ban/jail.local
    

    Paste this minimal + safe config:

    [DEFAULT]
    bantime  = 1h
    findtime = 10m
    maxretry = 5
    backend  = systemd
    ignoreip = 127.0.0.1/8 ::1
    
    [sshd]
    enabled = true
    port    = ssh
    logpath = %(sshd_log)s
    

    💡 backend = systemd is faster and more reliable on modern Pi OS.


    5. Restart Fail2Ban

    sudo systemctl restart fail2ban
    

    6. Confirm SSH jail is active

    sudo fail2ban-client status
    

    Expected:

    Jail list: sshd
    

    Check banned IPs:

    sudo fail2ban-client status sshd
    

    7. Reduce resource usage (IMPORTANT for 1 GB Pi)

    Edit:

    sudo nano /etc/fail2ban/fail2ban.conf
    

    Set:

    loglevel = INFO
    logtarget = /var/log/fail2ban.log
    

    Optional: disable polling (systemd handles logs efficiently).


    8. Enable log rotation (prevents SD wear)

    Check:

    sudo nano /etc/logrotate.d/fail2ban
    

    Recommended content:

    /var/log/fail2ban.log {
        weekly
        rotate 4
        compress
        missingok
        notifempty
    }
    

    9. Test Fail2Ban (safe test)

    From another machine:

    ssh wronguser@pi-ip
    

    Fail 5 times → IP gets banned.

    Unban yourself:

    sudo fail2ban-client set sshd unbanip YOUR_IP
    

    10. Optional hardening (strongly recommended)

    Disable SSH passwords

    sudo nano /etc/ssh/sshd_config
    

    Set:

    PasswordAuthentication no
    PermitRootLogin no
    

    Restart SSH:

    sudo systemctl restart ssh
    

    Summary (Pi-optimized)

    ✔ Low RAM usage
    ✔ SD-card friendly logging
    ✔ SSH protected
    ✔ systemd backend
    ✔ Safe upgrade-proof config

  • Running Mailu Behind Caddy with Let’s Encrypt (A Practical Guide)


    Self‑hosting email is one of those projects that sounds simple until you actually try it. Between TLS, reverse proxies, Autodiscover, and modern mail client expectations, it’s very easy to end up with redirect loops, broken SMTP, or certificates that load everywhere except where you need them.

    This post walks through a working, real‑world setup: running Mailu behind Caddy with Let’s Encrypt certificates (DNS‑01). It’s written from the perspective of what actually works, not just what looks correct on paper.

    All domains, paths, and identifiers in this post are examples.




    Why This Architecture

    Mail servers are special:

    They need TLS for web, IMAP, and SMTP

    They need predictable hostnames

    They must support Autodiscover for Outlook, Gmail, and mobile clients


    The architecture below solves all of that cleanly:

    Internet
       │
       ▼
    [Caddy :443]  ← public HTTPS
       │
       ▼
    [Mailu Front :443]  ← internal TLS
       │
       ├─ IMAP (993)
       ├─ SMTP (465)
       └─ Webmail / Admin

    Key design choice:

    > Caddy proxies to Mailu’s HTTPS port (443), not HTTP.



    That single decision avoids most redirect and TLS issues.




    Directory Layout (Example)

    Here’s a clean, anonymized layout that works well:

    /opt/mail/
    ├── mailu/
    │   ├── data/
    │   ├── mail/
    │   ├── dkim/
    │   ├── certs/
    │   │   ├── cert.pem
    │   │   └── key.pem
    │   ├── docker-compose.yml
    │   └── mailu.env

    ├── caddy/
    │   ├── Caddyfile
    │   └── docker-compose.yml

    └── letsencrypt/

    You can place this anywhere — /opt, /srv, or a mounted volume.




    Certificates: Let’s Encrypt with DNS‑01

    Mail servers benefit greatly from DNS‑01 validation:

    No open ports required

    Works with multiple subdomains

    Ideal behind proxies


    Requesting a Certificate (Example)

    certbot certonly \
      –dns-provider \
      -d mail.example.com \
      -d autodiscover.example.com \
      -d autoconfig.example.com

    After success:

    /etc/letsencrypt/live/mail.example.com/
    ├── fullchain.pem
    └── privkey.pem




    Making Certificates Available to Mailu

    Mailu expects certificates inside its own volume:

    cp /etc/letsencrypt/live/mail.example.com/fullchain.pem \
       /opt/mail/mailu/certs/cert.pem

    cp /etc/letsencrypt/live/mail.example.com/privkey.pem \
       /opt/mail/mailu/certs/key.pem

    chmod 644 /opt/mail/mailu/certs/cert.pem
    chmod 600 /opt/mail/mailu/certs/key.pem

    If these files are empty or truncated, IMAP and SMTP will fail immediately.




    Mailu Configuration (Minimal, Correct)

    In mailu.env:

    DOMAIN=example.com
    HOSTNAMES=mail.example.com,autodiscover.example.com,autoconfig.example.com
    TLS_FLAVOR=cert
    ENABLE_AUTODISCOVER=true

    WEB_ADMIN=/admin
    WEB_WEBMAIL=/webmail

    This ensures Mailu advertises the correct endpoints to clients.




    Caddy: The Reverse Proxy That Makes This Work

    Here’s the critical part.

    Caddyfile

    mail.example.com,
    autodiscover.example.com,
    autoconfig.example.com {

        @root path /
        redir @root /webmail 302

        reverse_proxy https://mailu-front:443 {
            transport http {
                tls_insecure_skip_verify
            }

            header_up Host {host}
            header_up X-Forwarded-Host {host}
            header_up X-Forwarded-Proto https
            header_up X-Forwarded-Port 443
        }
    }

    Why HTTPS → HTTPS Matters

    Mailu internally enforces HTTPS. If you proxy to port 80, nginx will continuously attempt to “upgrade” requests — even when the client is already using HTTPS.

    Result:

    Infinite 301 redirects

    Broken admin UI

    Webmail never loads


    Proxying to 443 avoids this entirely.




    Starting the Stack

    cd /opt/mail/mailu
    docker compose up -d

    cd /opt/mail/caddy
    docker compose up -d

    Confirm everything is running:

    docker ps




    Verifying the Web Interfaces

    curl -Ik https://mail.example.com/admin/
    curl -Ik https://mail.example.com/webmail/

    Expected:

    No redirect loops

    Login page loads





    Verifying Mail Protocols

    IMAP (993)

    openssl s_client -connect mail.example.com:993 \
      -servername mail.example.com -brief </dev/null

    SMTP (465)

    openssl s_client -connect mail.example.com:465 \
      -servername mail.example.com -brief </dev/null

    Modern Mailu deployments often do not support STARTTLS on 587. Port 465 is the correct choice.




    Autodiscover Test

    curl -X POST https://autodiscover.example.com/autodiscover/autodiscover.xml \
      -H “Content-Type: text/xml” \
      –data ‘<Autodiscover xmlns=”http://schemas.microsoft.com/exchange/autodiscover/outlook/requestschema/2006″>
      <Request>
        <EMailAddress>user@example.com</EMailAddress>
        <AcceptableResponseSchema>
          http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a
        </AcceptableResponseSchema>
      </Request>
    </Autodiscover>’

    Clients should receive IMAP and SMTP settings automatically.




    Automatic Certificate Renewal

    certbot renew –deploy-hook “
      docker restart mailu-front \
      docker restart mailu-imap \
      docker restart mailu-smtp


    This ensures mail protocols reload certificates correctly.




    Lessons Learned

    Proxying Mailu via HTTP causes redirect loops

    Empty certificate files break IMAP/SMTP instantly

    Port 465 is now the safest SMTP option

    DNS‑01 validation is ideal for mail servers





    Final Thoughts

    Mail servers don’t fail loudly — they fail subtly. A single redirect, missing TLS flag, or incorrect port can break everything downstream.

    This setup has proven reliable, debuggable, and future‑proof. If you’re self‑hosting mail today, this architecture will save you hours of frustration.




    All paths and domains shown are examples. Replace them with values appropriate to your environment.

Secret Link