Category: Linux Server

  • N8N Project for My production Server

    Production Monitoring & Security Automation Runbook

    Purpose

    This runbook describes how to operate, monitor, and respond to events generated by the company’s production automation stack:

    • n8n (orchestration)
    • AdGuard DNS (Primary & Secondary Raspberry Pi)
    • Fail2Ban
    • Uptime Kuma
    • Slack (alerting)
    • Omada Controller (network devices)

    It is written so that any on-call engineer can safely respond to alerts without deep system knowledge.


    System Overview

    What this system does

    • Monitors DNS behavior on two AdGuard servers (Pi1 = Primary, Pi2 = Secondary)
    • Detects possible DNS abuse / attacks using query heuristics
    • Automatically blocks malicious IPs in AdGuard (when enabled)
    • Monitors uptime of both DNS servers
    • Pushes health heartbeats to Uptime Kuma
    • Receives Fail2Ban ban/unban events from multiple hosts
    • Receives Omada controller events (AP, gateway, switch up/down)
    • Sends actionable alerts to Slack

    What it does NOT do

    • It does not permanently blacklist IPs without review
    • It does not modify firewall rules (DNS-layer only)
    • It does not auto-restart servers

    Normal Operation (Healthy State)

    Expected behavior

    • Cron runs every minute
    • Slack is quiet most of the time
    • Uptime Kuma shows:
      • Pi1 Uptime: UP
      • Pi2 Uptime: UP
      • DNS Status: NORMAL

    Normal Slack messages

    • ✅ DNS NORMAL (baseline)
    • ✅ DNS OK / RECOVERED
    • ✅ Fail2Ban UNBANNED
    • ℹ️ Omada informational events

    No action is required in these cases.


    Alert Types & Response Actions

    🚨 POSSIBLE DNS ATTACK

    Meaning

    • One client is responsible for an abnormally high percentage of DNS queries
    • Triggered when:
      • ≥ 80% of recent queries OR
      • ≥ 400 queries in sample window

    Automatic actions

    • AdGuard auto-block may already be applied
    • IP reputation (IPinfo) is attached to the alert

    Required response (step-by-step)

    1. Open the Slack alert
    2. Review:
      • Attacker IP
      • Client name (if known)
      • Organization / ASN
    3. Log into the affected AdGuard server
    4. Open Query Log
    5. Confirm traffic pattern matches alert
    6. If legitimate client:
      • Remove IP from disallowed_clients
      • Add client to DNS whitelist in n8n
    7. If malicious:
      • No action needed (auto-block handled it)

    Escalation

    • Repeated attacks from different IPs → notify network/security team

    ✅ DNS OK / RECOVERED

    Meaning

    • DNS traffic has returned to normal

    Action

    • None required

    🔴 / 🚨 UPTIME DOWN

    Meaning

    • DNS server is unreachable or returning bad HTTP status

    Response steps

    1. Check Uptime Kuma for confirmation
    2. Attempt to reach host:
      • Ping
      • HTTPS access
    3. If unreachable:
      • Check power
      • Check network connectivity
    4. Review system logs if accessible
    5. Restart service/server if required

    Escalation

    • If downtime > SLA threshold, notify management

    🚫 Fail2Ban BANNED

    Meaning

    • Fail2Ban blocked an IP due to repeated authentication failures

    Automatic actions

    • IP already blocked at service level
    • Geo/IP data added automatically

    Response steps

    1. Review IP reputation in Slack
    2. Confirm jail name (sshd, nginx, etc.)
    3. If internal or known IP:
      • Manually unban
      • Adjust Fail2Ban rules if needed
    4. If external/malicious:
      • No action required

    🚨 Omada Device DOWN

    Meaning

    • AP, gateway, or switch disconnected

    Response steps

    1. Identify device and site in Slack alert
    2. Check Omada Controller status
    3. Verify power and uplink
    4. If multiple devices affected:
      • Suspect upstream outage

    Environment & Configuration

    Required environment variables (n8n)

    • F2B_TOKEN
    • IPINFO_TOKEN
    • KUMA_PI1_UPTIME_URL
    • KUMA_PI1_DNS_URL
    • KUMA_PI2_UPTIME_URL
    • KUMA_PI2_DNS_URL

    Webhook endpoints

    • /fail2ban-pi1
    • /fail2ban-pi2
    • /OmadaController
    • /JoeOmadaTPlink

    Maintenance & Safe Changes

    Before making changes

    • Disable auto-block if testing
    • Clone workflow for testing
    • Verify Slack output formatting

    After changes

    • Manually trigger workflow
    • Confirm:
      • No duplicate Slack alerts
      • Kuma heartbeats still flow

    Break-Glass (Emergency)

    If automation behaves incorrectly:

    1. Disable the n8n workflow
    2. Remove IPs from AdGuard block list
    3. Notify security/network team
    4. Document incident

    Ownership

    • System owner: IT / Network Team
    • Primary contact: IT Manager
    • Slack channel: Monitoring / Security Alerts

  • Running Mailu Behind Caddy with Let’s Encrypt (A Practical Guide)


    Self‑hosting email is one of those projects that sounds simple until you actually try it. Between TLS, reverse proxies, Autodiscover, and modern mail client expectations, it’s very easy to end up with redirect loops, broken SMTP, or certificates that load everywhere except where you need them.

    This post walks through a working, real‑world setup: running Mailu behind Caddy with Let’s Encrypt certificates (DNS‑01). It’s written from the perspective of what actually works, not just what looks correct on paper.

    All domains, paths, and identifiers in this post are examples.




    Why This Architecture

    Mail servers are special:

    They need TLS for web, IMAP, and SMTP

    They need predictable hostnames

    They must support Autodiscover for Outlook, Gmail, and mobile clients


    The architecture below solves all of that cleanly:

    Internet
       │
       ▼
    [Caddy :443]  ← public HTTPS
       │
       ▼
    [Mailu Front :443]  ← internal TLS
       │
       ├─ IMAP (993)
       ├─ SMTP (465)
       └─ Webmail / Admin

    Key design choice:

    > Caddy proxies to Mailu’s HTTPS port (443), not HTTP.



    That single decision avoids most redirect and TLS issues.




    Directory Layout (Example)

    Here’s a clean, anonymized layout that works well:

    /opt/mail/
    ├── mailu/
    │   ├── data/
    │   ├── mail/
    │   ├── dkim/
    │   ├── certs/
    │   │   ├── cert.pem
    │   │   └── key.pem
    │   ├── docker-compose.yml
    │   └── mailu.env

    ├── caddy/
    │   ├── Caddyfile
    │   └── docker-compose.yml

    └── letsencrypt/

    You can place this anywhere — /opt, /srv, or a mounted volume.




    Certificates: Let’s Encrypt with DNS‑01

    Mail servers benefit greatly from DNS‑01 validation:

    No open ports required

    Works with multiple subdomains

    Ideal behind proxies


    Requesting a Certificate (Example)

    certbot certonly \
      –dns-provider \
      -d mail.example.com \
      -d autodiscover.example.com \
      -d autoconfig.example.com

    After success:

    /etc/letsencrypt/live/mail.example.com/
    ├── fullchain.pem
    └── privkey.pem




    Making Certificates Available to Mailu

    Mailu expects certificates inside its own volume:

    cp /etc/letsencrypt/live/mail.example.com/fullchain.pem \
       /opt/mail/mailu/certs/cert.pem

    cp /etc/letsencrypt/live/mail.example.com/privkey.pem \
       /opt/mail/mailu/certs/key.pem

    chmod 644 /opt/mail/mailu/certs/cert.pem
    chmod 600 /opt/mail/mailu/certs/key.pem

    If these files are empty or truncated, IMAP and SMTP will fail immediately.




    Mailu Configuration (Minimal, Correct)

    In mailu.env:

    DOMAIN=example.com
    HOSTNAMES=mail.example.com,autodiscover.example.com,autoconfig.example.com
    TLS_FLAVOR=cert
    ENABLE_AUTODISCOVER=true

    WEB_ADMIN=/admin
    WEB_WEBMAIL=/webmail

    This ensures Mailu advertises the correct endpoints to clients.




    Caddy: The Reverse Proxy That Makes This Work

    Here’s the critical part.

    Caddyfile

    mail.example.com,
    autodiscover.example.com,
    autoconfig.example.com {

        @root path /
        redir @root /webmail 302

        reverse_proxy https://mailu-front:443 {
            transport http {
                tls_insecure_skip_verify
            }

            header_up Host {host}
            header_up X-Forwarded-Host {host}
            header_up X-Forwarded-Proto https
            header_up X-Forwarded-Port 443
        }
    }

    Why HTTPS → HTTPS Matters

    Mailu internally enforces HTTPS. If you proxy to port 80, nginx will continuously attempt to “upgrade” requests — even when the client is already using HTTPS.

    Result:

    Infinite 301 redirects

    Broken admin UI

    Webmail never loads


    Proxying to 443 avoids this entirely.




    Starting the Stack

    cd /opt/mail/mailu
    docker compose up -d

    cd /opt/mail/caddy
    docker compose up -d

    Confirm everything is running:

    docker ps




    Verifying the Web Interfaces

    curl -Ik https://mail.example.com/admin/
    curl -Ik https://mail.example.com/webmail/

    Expected:

    No redirect loops

    Login page loads





    Verifying Mail Protocols

    IMAP (993)

    openssl s_client -connect mail.example.com:993 \
      -servername mail.example.com -brief </dev/null

    SMTP (465)

    openssl s_client -connect mail.example.com:465 \
      -servername mail.example.com -brief </dev/null

    Modern Mailu deployments often do not support STARTTLS on 587. Port 465 is the correct choice.




    Autodiscover Test

    curl -X POST https://autodiscover.example.com/autodiscover/autodiscover.xml \
      -H “Content-Type: text/xml” \
      –data ‘<Autodiscover xmlns=”http://schemas.microsoft.com/exchange/autodiscover/outlook/requestschema/2006″>
      <Request>
        <EMailAddress>user@example.com</EMailAddress>
        <AcceptableResponseSchema>
          http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a
        </AcceptableResponseSchema>
      </Request>
    </Autodiscover>’

    Clients should receive IMAP and SMTP settings automatically.




    Automatic Certificate Renewal

    certbot renew –deploy-hook “
      docker restart mailu-front \
      docker restart mailu-imap \
      docker restart mailu-smtp


    This ensures mail protocols reload certificates correctly.




    Lessons Learned

    Proxying Mailu via HTTP causes redirect loops

    Empty certificate files break IMAP/SMTP instantly

    Port 465 is now the safest SMTP option

    DNS‑01 validation is ideal for mail servers





    Final Thoughts

    Mail servers don’t fail loudly — they fail subtly. A single redirect, missing TLS flag, or incorrect port can break everything downstream.

    This setup has proven reliable, debuggable, and future‑proof. If you’re self‑hosting mail today, this architecture will save you hours of frustration.




    All paths and domains shown are examples. Replace them with values appropriate to your environment.

  • How to Install WordPress on Portainer with Caddy Proxy (Automatic HTTPS)

    Running WordPress behind a modern, automatic HTTPS reverse proxy is easier than ever using Docker, Portainer, and Caddy.

    In this guide, you will deploy:

    • Caddy – reverse proxy and automatic TLS (Let’s Encrypt)
    • WordPress – application container
    • MariaDB – database container
    • Portainer – web UI to manage your Docker stack

    By the end, you will have a secure, production-ready WordPress site accessible at your domain with automatic certificate management and clean container isolation.


    Architecture Overview

    You will run three services:

    • Caddy
      • Public-facing
      • Listens on ports 80 and 443
      • Automatically provisions and renews HTTPS certificates
      • Reverse proxies traffic to WordPress
    • WordPress
      • Only reachable internally
      • Connected to Caddy via a shared network
      • Connected to MariaDB via a private network
    • MariaDB
      • Internal-only
      • Not exposed to the public internet

    Networking Model

    We use two Docker networks:

    • web → Shared by Caddy and WordPress (public-facing traffic)
    • wp → Private network between WordPress and MariaDB

    We also use persistent Docker volumes:

    • caddy_data → TLS certificates and Caddy state
    • db_data → MariaDB database files
    • wp_data → WordPress files and uploads

    This ensures container restarts or updates do not erase data.


    Prerequisites

    Before deploying anything, confirm the following:

    1. Domain & DNS

    Choose your canonical hostname:

    • example.com
    • or www.example.com

    Create DNS records:

    • A record → your server’s public IPv4
    • AAAA record (optional) → your server’s IPv6

    Confirm DNS resolves correctly:

    dig example.com
    

    Do not continue until DNS points to your server.


    2. Server Requirements

    • Linux server (Ubuntu/Debian recommended)
    • Public IP address
    • Open ports:
      • 80/tcp
      • 443/tcp
      • 9443/tcp (Portainer UI)

    Make sure no existing service (Apache/Nginx) is using ports 80 or 443.


    Step 1 – Install Docker

    curl -fsSL https://get.docker.com -o get-docker.sh
    sudo sh get-docker.sh
    sudo usermod -aG docker $USER
    newgrp docker
    

    Verify installation:

    docker --version
    

    Step 2 – Install Portainer

    Create a volume:

    docker volume create portainer_data
    

    Run Portainer:

    docker run -d \
      -p 8000:8000 \
      -p 9443:9443 \
      --name portainer \
      --restart=always \
      -v /var/run/docker.sock:/var/run/docker.sock \
      -v portainer_data:/data \
      portainer/portainer-ce:latest
    

    Access Portainer:

    https://YOUR_SERVER_IP:9443
    

    Complete the admin setup and select the local Docker environment.


    Step 3 – Create Docker Networks and Volumes

    Create networks:

    docker network create web
    docker network create wp
    

    Create persistent volumes:

    docker volume create caddy_data
    docker volume create db_data
    docker volume create wp_data
    

    These must exist before deploying the stack.


    Step 4 – Create the Caddyfile

    Create directory:

    sudo mkdir -p /opt/caddy
    sudo nano /opt/caddy/Caddyfile
    

    Example configuration (redirect www → apex):

    {
      email you@example.com
    }
    
    www.example.com {
      redir https://example.com{uri} permanent
    }
    
    example.com {
      encode zstd gzip
    
      header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        X-Content-Type-Options "nosniff"
        Referrer-Policy "strict-origin-when-cross-origin"
      }
    
      reverse_proxy wordpress:80 {
        header_up X-Forwarded-Proto {scheme}
        header_up X-Forwarded-Host {host}
      }
    }
    

    Replace:

    • example.com
    • you@example.com

    Step 5 – Deploy WordPress Stack in Portainer

    Go to:

    Portainer → Stacks → Add Stack

    Name it:

    wordpress-with-caddy
    

    Paste this Docker Compose file:

    version: "3.8"
    
    services:
      caddy:
        image: caddy:2
        container_name: caddy
        restart: unless-stopped
        ports:
          - "80:80"
          - "443:443"
        volumes:
          - caddy_data:/data
          - /opt/caddy/Caddyfile:/etc/caddy/Caddyfile:ro
        networks:
          - web
    
      db:
        image: mariadb:11
        container_name: wp-db
        restart: unless-stopped
        environment:
          MARIADB_DATABASE: wordpress
          MARIADB_USER: wp
          MARIADB_PASSWORD: CHANGE_ME_STRONG
          MARIADB_ROOT_PASSWORD: CHANGE_ME_VERY_STRONG
        volumes:
          - db_data:/var/lib/mysql
        networks:
          - wp
    
      wordpress:
        image: wordpress:6-php8.2-apache
        container_name: wordpress
        restart: unless-stopped
        depends_on:
          - db
        environment:
          WORDPRESS_DB_HOST: db:3306
          WORDPRESS_DB_USER: wp
          WORDPRESS_DB_PASSWORD: CHANGE_ME_STRONG
          WORDPRESS_DB_NAME: wordpress
        volumes:
          - wp_data:/var/www/html
        networks:
          - web
          - wp
    
    volumes:
      caddy_data:
        external: true
      db_data:
        external: true
      wp_data:
        external: true
    
    networks:
      web:
        external: true
      wp:
        external: true
    

    Replace all passwords before deploying.

    Click Deploy the Stack.


    Step 6 – Verify Certificate Issuance

    Go to:

    Containers → caddy → Logs

    You should see successful ACME certificate issuance.

    If you see errors:

    • Confirm DNS is correct
    • Confirm ports 80 and 443 are open
    • Restart only the Caddy container after fixing issues

    Step 7 – Fix WordPress Behind Reverse Proxy (Important)

    Because Caddy terminates TLS, WordPress must be told it is running behind HTTPS.

    Edit wp-config.php (inside the wp_data volume).

    Add this above:

    /* That's all, stop editing! Happy publishing. */
    
    if (
      (!empty($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] === 'https')
      || (!empty($_SERVER['HTTP_X_FORWARDED_SSL']) && $_SERVER['HTTP_X_FORWARDED_SSL'] === 'on')
    ) {
      $_SERVER['HTTPS'] = 'on';
      $_SERVER['SERVER_PORT'] = 443;
    }
    
    define('FORCE_SSL_ADMIN', true);
    

    This prevents:

    • Mixed content errors
    • Login redirect loops
    • Secure cookie problems

    Step 8 – Complete WordPress Setup

    Visit:

    https://example.com
    

    Complete the installer:

    • Choose language
    • Set site title
    • Create strong admin credentials

    After installation:

    • Go to Settings → General
    • Confirm both URLs use HTTPS
    • Go to Settings → Permalinks → Save

    Upload a test image and confirm it loads over HTTPS.


    Backups

    Back up both:

    Database

    docker exec wp-db sh -c 'exec mysqldump -u wp -p"$MARIADB_PASSWORD" wordpress' > wp-$(date +%F).sql
    

    WordPress files

    Archive the wp_data volume regularly.


    Hardening Recommendations

    • Use strong, unique DB passwords
    • Keep Docker images updated
    • Limit exposed ports (only 80/443 publicly)
    • Enable 2FA in WordPress
    • Remove unused plugins/themes
    • Consider disabling XML-RPC if unused

    Troubleshooting

    Caddy cannot issue certificates

    • DNS incorrect
    • Ports 80/443 blocked
    • Another service using 80/443

    502 / 504 Errors

    • WordPress container not running
    • Network misconfiguration
    • DB credentials incorrect

    Mixed Content Warnings

    • Hardcoded http links in theme or database
    • Missing proxy HTTPS snippet

    Final Result

    You now have:

    • Automatic HTTPS via Caddy
    • Isolated database
    • Persistent volumes
    • Easy stack management via Portainer
    • Clean, production-ready WordPress deployment

    This setup is simple, secure, and scalable. You can extend it with:

    • Redis caching
    • Staging stacks
    • Automated backups
    • CI/CD deployments

Secret Link