today

AGENTS.md

@@ -0,0 +1,308 @@
+# AGENTS.md — 4server Project Reference
+
+## What This Is
+
+**4server** is a self-hosted infrastructure management platform. It runs a Docker-based Alpine Linux control container locally and manages a fleet of remote Alpine Linux servers. Servers host multi-tenant Odoo (17/18/19) and N8N instances as Docker containers, all behind a Traefik reverse proxy with auto-TLS.
+
+---
+
+## Repository Layout
+
+```
+4server/
+├── alpine/             # Control container (Dockerfile + entrypoint scripts)
+│   ├── Dockerfile      # Alpine image with pssh, borg, sshfs, nvim, cryptsetup, etc.
+│   ├── config          # SSH config for all managed hosts (user: 4server)
+│   ├── known_hosts     # Pre-populated SSH known_hosts
+│   ├── rex             # Run a command on ALL hosts via pssh
+│   ├── template        # Deploy a template file to ALL hosts (with variable substitution)
+│   ├── create_volume   # Create and LUKS-format an encrypted volume
+│   └── mount_volume    # Mount an encrypted volume
+│
+├── app/                # Mounted as /app inside the control container
+│   ├── host_vars/      # Per-host SSH keys, env files, and hosts list
+│   │   ├── hosts       # Plain-text list of hostnames for pssh/prsync
+│   │   ├── create      # Script to generate Ed25519 SSH key pairs
+│   │   └── {host}/     # Directory per host: {host} (private key), {host}.pub, {host}.env
+│   │
+│   ├── sbin/           # Scripts deployed to /4server/sbin/ on every server
+│   │   ├── api         # FastAPI REST API (Python 3, uvicorn) — server management API
+│   │   ├── helpers     # Bash lib: get_contract_info() — loads UUID data from SQLite
+│   │   ├── startContainer    # Dispatch start by UUID type
+│   │   ├── stopContainer     # Stop a container
+│   │   ├── nukeContainer     # Destroy container (must be stopped first)
+│   │   ├── backupContainer   # Dispatch backup by UUID type
+│   │   ├── backupAll         # Backup all running NNN-NNN containers
+│   │   ├── borgpush          # Push /BACKUP/current* files to BorgBase
+│   │   ├── stats             # JSON: uptime, top, cgroup, mounts
+│   │   ├── audit             # JSON: full Docker + OS audit
+│   │   ├── checkCalls        # OpenRC service: monitors API calls
+│   │   ├── cleanTmp          # OpenRC service: cleans /4server/tmp
+│   │   ├── cpu               # OpenRC service: CPU monitoring
+│   │   ├── gitPull/gitRevert # Git operations inside containers
+│   │   ├── migrate           # Data migration helpers
+│   │   ├── sql / psql        # PostgreSQL query helpers
+│   │   ├── contractInfo      # Print contract info for a UUID
+│   │   ├── getContainers     # List containers
+│   │   ├── pullAllContainers # docker pull on all containers
+│   │   ├── restartContainers # Restart all containers
+│   │   ├── odoo_passwd       # Reset Odoo user password
+│   │   ├── start/            # Type-specific start scripts (n8n, ODOO_17/18/19)
+│   │   ├── backup/           # Type-specific backup scripts (N8N, ODOO)
+│   │   ├── nuke/             # Type-specific nuke scripts (n8n, ODOO_19)
+│   │   ├── ODOO_19/          # Odoo 19 admin tools (restore, shell, updateModules, audit, etc.)
+│   │   └── N8N/              # N8N admin tools (currently empty)
+│   │
+│   ├── templates/      # Config templates deployed via the `template` command
+│   │   ├── docker-compose.yml   # Base infra: postgres (beedb) + traefik
+│   │   ├── traefik.yaml         # Traefik config
+│   │   ├── repositories         # APK repository list
+│   │   ├── hosts                # /etc/hosts
+│   │   ├── hostname             # /etc/hostname
+│   │   ├── extlinux.conf        # Bootloader (variants for DO, Hostinger)
+│   │   ├── nebula/              # Nebula VPN: config.yml, ca.crt, host.crt, host.key
+│   │   ├── init.d/              # OpenRC service files: nebula, api, checkCalls, cleanTmp, cpu, ping_service
+│   │   ├── ssh/                 # SSH authorized_keys (id_ed25519.pub)
+│   │   ├── etc/doas.d/          # doas config (4server.conf)
+│   │   └── .profile             # Shell profile for 4server user
+│   │
+│   ├── etc/traefik/certs/  # TLS certificates distributed to servers
+│   ├── vault/              # Encrypted secrets (not committed)
+│   │
+│   ├── onboarding      # New server bootstrap (Alpine packages, Nebula VPN, 4server user)
+│   ├── hardening       # Lock down SSH (no root, no password login)
+│   ├── update          # Full server update (packages, Docker infra, API, Traefik)
+│   ├── update_sbin     # Deploy /app/sbin/ to servers + restart services
+│   ├── update_alpine   # APK update/upgrade only
+│   ├── restart         # Restart core services
+│   ├── backup_prune    # Prune old Borg archives
+│   ├── migrate         # One-off DB migrations
+│   └── download_sbin   # Download binary tools (nebula, etc.)
+│
+├── exchange/           # Shared volume between host and control container (file drop)
+├── docker-compose.yaml # Builds + runs the Alpine control container
+├── mount               # Helper to mount exchange volume
+└── start               # Start the control container
+```
+
+---
+
+## Servers (Managed Fleet)
+
+| Alias      | IP              | Port | Notes              |
+|------------|-----------------|------|--------------------|
+| dev        | 192.168.9.221   | 2222 | Development server |
+| manchester | 192.168.9.20    | 22   |                    |
+| boston     | 192.168.9.16    | 22   |                    |
+| mumbai     | 192.168.9.17    | 22   |                    |
+| meppel     | 192.168.9.21    | 22   |                    |
+| sydney     | 192.168.9.22    | 22   |                    |
+| saopaulo   | 192.168.9.11    | 22   |                    |
+
+SSH user on all servers: `4server`  
+Key per host: `/app/host_vars/{host}/{host}` (Ed25519)  
+Active hosts file: `/app/host_vars/hosts` (currently: `sydney2`)
+
+---
+
+## UUID Format
+
+Every managed container has a UUID structured as: `{server}-{type}-{random}`
+
+| Segment | Meaning      | Known values                              |
+|---------|-------------|-------------------------------------------|
+| `aaa`   | Server       | `001` = manchester, `002` = boston        |
+| `bbb`   | Image type   | `001` = N8N, `002` = Odoo 18, `003` = Odoo 19, `004` = Odoo 17 |
+
+The second segment (`bbb`) is used by `startContainer`, `nukeContainer`, `backupContainer` to dispatch to the right type-specific script.
+
+---
+
+## Core Tools (inside the Alpine control container)
+
+### `rex <command>`
+Runs a shell command on **all hosts** in `/app/host_vars/hosts` via `pssh`.  
+Example: `rex doas apk update`
+
+### `prsync -h /app/host_vars/hosts -avz <src> <dst>`
+Parallel rsync to all hosts. Used to deploy `/app/sbin/` and certs.
+
+### `template <localfile> <remotefile>`
+Renders `{{KEY}}` placeholders in `localfile` and copies it to `remotefile` on every host.  
+Available template variables (loaded from `host_vars/{host}/{host}.env` + key files):
+- `NEBULA_CA`, `NEBULA_CRT`, `NEBULA_KEY`
+- `API_KEY`, `HOSTNAME`
+- `SSH_PRIVATE`, `SSH_PUBLIC`
+
+---
+
+## Server-Side Architecture (on each managed server)
+
+```
+/4server/
+├── sbin/               # All management scripts (deployed from app/sbin/)
+├── data/
+│   ├── contracts.db    # SQLite: container contracts table
+│   ├── log/            # Logs for all services
+│   ├── {UUID}/         # Per-container data directory
+│   │   ├── odoo/       # Odoo filestore
+│   │   ├── git/        # Git repos
+│   │   ├── git-server/ # Bare git repos exposed via SSH
+│   │   ├── logs/
+│   │   ├── config/
+│   │   ├── cc/         # Control channel (e.g. backup trigger file)
+│   │   ├── etc/        # domain file, gitpath, etc.
+│   │   ├── install/
+│   │   └── .ssh/       # Container SSH keys
+│   ├── traefik/etc/    # traefik.yaml + TLS certs
+│   └── postgres/       # PostgreSQL data + pg_hba.conf
+├── tmp/                # Temporary files (cleaned by cleanTmp service)
+└── docker-compose.yml  # Base infra (beedb + traefik)
+
+/BACKUP/{UUID}/         # Per-container backup files (borgpush picks up current*)
+```
+
+### Persistent Docker Services (base infra)
+| Container | Image         | Role                          | IP (internal)  |
+|-----------|---------------|-------------------------------|----------------|
+| `beedb`   | postgres:16   | Shared PostgreSQL for all Odoo| 10.5.0.200     |
+| `traefik` | traefik:3.6   | Reverse proxy, HTTPS (Let's Encrypt) | —         |
+
+Docker network: `4server_4projects` (bridge, `10.5.0.0/16`)
+
+### Per-Container Docker Services
+- Each UUID is a `docker run` container on the host's Docker network
+- SSH port: `CONTAINERDBID + 2200` (forwarded to the host)
+- Traefik routing via Docker labels (`traefik.http.routers.{UUID}.*`)
+- Default domain: `{UUID}.odoo4projects.com`; optional custom domains from `/4server/data/{UUID}/etc/domain`
+
+---
+
+## Data Model: `contracts.db`
+
+Table: `containers`
+
+| Column         | Description                                        |
+|----------------|----------------------------------------------------|
+| `UUID`         | Container identifier (aaa-bbb-... format)          |
+| `id`           | Auto-increment row ID (`CONTAINERDBID`)            |
+| `image`        | Image type code                                    |
+| `status`       | Container status                                   |
+| `expires`      | Contract expiry date                               |
+| `created`      | Creation timestamp                                 |
+| `secret_list`  | Backslash-separated `key=value` pairs (passwords)  |
+| `git`          | Git integration config                             |
+| `backup_slots` | Number of backup slots                             |
+| `hdd`          | Disk quota                                         |
+| `workers`      | Worker count                                       |
+| `domains_slots`| Number of allowed custom domains                   |
+| `domains`      | Active custom domains (comma-separated)            |
+| `affiliate`    | JSON: UTM tracking data                            |
+
+`secret_list` parsing: `key=value` pairs separated by `\`. Key `psql` = Odoo DB password (`ODOO_DB_PASSWORD`).
+
+---
+
+## Management API (`/4server/sbin/api`)
+
+FastAPI Python 3 app, served by uvicorn, managed by OpenRC (`api` service).
+
+- Auth: `X-API-Key` header (value from `/etc/4server` env file, key `API_KEY`)
+- Version: `0.0.9`
+
+Key endpoints (all protected):
+| Endpoint | Description |
+|----------|-------------|
+| `GET /containers` | List all containers from SQLite |
+| `POST /container/update` | Create/update a container record |
+| `POST /container/start` | Start a container by UUID |
+| `POST /container/stop` | Stop a container by UUID |
+| `POST /container/nuke` | Destroy a container |
+| `GET /container/info` | Full container info + Docker inspect |
+| `POST /container/bump` | Bump/restart a container |
+| `GET /container/quota` | Disk quota usage |
+| `GET /images` | List available Docker images |
+| `GET /cpu` | CPU log |
+| `GET /stats` | Server runtime stats (JSON) |
+| `GET /audit` | Full Docker + OS audit (JSON) |
+| `POST /git` | Git pull/revert inside container |
+| `POST /git/key` | Add SSH key to git-server |
+| `GET /odoo/audit` | Odoo-specific audit |
+| `GET /odoo/log` | Summarised Odoo log (last N lines, errors/warnings) |
+| `POST /backup/import` | Import a backup archive |
+| `POST /backup/move` | Move a backup file |
+| `POST /backup/check` | Check backup integrity |
+| `POST /backup/all` | Trigger backup of all containers |
+| `POST /backup/borgpush` | Push backups to BorgBase |
+
+---
+
+## Backup
+
+- **Per-container**: `backupContainer <UUID>` → dispatches to `backup/N8N` or `backup/ODOO`
+- **All containers**: `backupAll` — touches `/4server/data/{UUID}/cc/backup` for each running NNN-NNN container
+- **Off-site**: `borgpush` → BorgBase (`e7e9h45y.repo.borgbase.com`) via SSH key `~/.ssh/borg-backup`
+- **Pruning**: `backup_prune` — keep 7 daily / 8 weekly / 12 monthly
+
+---
+
+## Security Model
+
+- **Privilege escalation**: `doas` (not sudo). Config at `/etc/doas.d/4server.conf`
+- **SSH**: Key-only (`PasswordAuthentication no`, `PermitRootLogin no`)
+- **API**: Header-based API key (`X-API-Key`)
+- **Secrets**: Stored in `secret_list` column of SQLite (base64-safe `\`-separated pairs)
+- **VPN**: Nebula overlay mesh between all servers (certs per host in `host_vars/{host}/`)
+- **TLS**: Traefik with Let's Encrypt production resolver
+
+---
+
+## Workflow Reference
+
+### Add a new server
+1. Add SSH key pair: `cd app/host_vars && ./create {hostname}`
+2. Create `host_vars/{host}/{host}.env` with `HOSTNAME`, `API_KEY`, `NEBULA_CRT`, `NEBULA_KEY`
+3. Add host to `host_vars/hosts`
+4. Add SSH config block to `alpine/config`
+5. Run `onboarding` from inside the control container
+6. Run `hardening` to lock down SSH
+
+### Deploy code changes to servers
+```bash
+# Inside control container (/app)
+./update_sbin        # deploy sbin/ scripts + restart api/checkCalls
+```
+
+### Deploy a config template
+```bash
+template templates/my.conf /etc/my.conf
+```
+
+### Run a command on all servers
+```bash
+rex doas rc-service api restart
+```
+
+### Start/stop a specific container (from server)
+```bash
+startContainer <UUID>
+stopContainer <UUID>
+```
+
+---
+
+## Key Technologies
+
+| Layer | Technology |
+|-------|------------|
+| Control container | Alpine Linux + Docker Compose |
+| Server OS | Alpine Linux (OpenRC init system) |
+| Reverse proxy | Traefik v3.6 |
+| Database | PostgreSQL 16 (shared, per-server) |
+| App runtime | Odoo 17/18/19, N8N |
+| Management API | Python 3 / FastAPI / uvicorn |
+| Backup | BorgBackup → BorgBase |
+| VPN | Nebula mesh |
+| Parallel SSH | pssh / prsync |
+| Container secrets | SQLite (`contracts.db`) |
+| SSH auth | Ed25519 keys per host |

app/sbin/api

@@ -501,7 +501,7 @@ async def get_odoo_log_summary(uuid: str):
     # --- Main logic ---
     try:
         # Last 500 lines from Odoo log
-        last_500_lines = read_last_lines(odoo_log_file, 500)
+        last_500_lines = read_last_lines(odoo_log_file, 5000)
         important_odoo_lines = [
             line for line in last_500_lines if is_important_line(line)
         ]

app/sbin/checkFilestore

@@ -0,0 +1,58 @@
+#!/bin/bash
+# checkFilestore <UUID> — compare container filestore vs host disk via md5
+
+check_one() {
+local UUID="$1"
+
+FILESTORE_HOST="/4server/data/$UUID/odoo/odoo-web-data/filestore/$UUID"
+
+[[ -d "$FILESTORE_HOST" ]] || { echo "ERROR: host filestore not found: $FILESTORE_HOST"; exit 2; }
+doas docker ps --format "{{.Names}}" | grep -q "^${UUID}$" \
+    || { echo "ERROR: container $UUID is not running"; exit 2; }
+
+# Read data_dir from odoo.conf inside the container, fall back to default
+DATA_DIR=$(doas docker exec "$UUID" sh -c \
+    'for f in /etc/odoo.conf /etc/odoo/odoo.conf /opt/odoo/etc/odoo.conf; do
+         grep -E "^data_dir" "$f" 2>/dev/null | cut -d= -f2- | xargs && break
+     done')
+DATA_DIR="${DATA_DIR:-/home/odoo/.local/share/Odoo}"
+FILESTORE_CONTAINER="$DATA_DIR/filestore/$UUID"
+MOUNT_HOST=$(doas docker inspect "$UUID" --format '{{json .Mounts}}' \
+    | jq -r --arg dest "$DATA_DIR" '.[] | select(.Type=="bind") | select(.Destination==$dest) | .Source')
+echo "ODOO USES        : $FILESTORE_CONTAINER"
+echo "Docker maps to   : $DATA_DIR"
+echo ""
+echo "Docker mapped to : ${MOUNT_HOST:-<no matching mount found>}"
+
+echo "Hashing container filestore..."
+MD5_CONTAINER=$(doas docker exec "$UUID" sh -c \
+    "find $FILESTORE_CONTAINER -type f | sort | xargs md5sum 2>/dev/null | awk '{print \$1}'" | md5sum | cut -d' ' -f1)
+
+echo "Hashing host filestore..."
+MD5_HOST=$(find "$FILESTORE_HOST" -type f | sort | xargs md5sum 2>/dev/null | awk '{print $1}' | md5sum | cut -d' ' -f1)
+
+echo "Container : $MD5_CONTAINER"
+echo "Host      : $MD5_HOST"
+
+[[ "$MD5_CONTAINER" == "$MD5_HOST" ]] && echo "OK: filestores match" || { echo "FAIL: mismatch"; return 3; }
+}
+
+if [[ -n "${1-}" ]]; then
+    check_one "$1"
+else
+    for dir in /4server/data/00*/; do
+        UUID=$(basename "$dir")
+        result=$(check_one "$UUID" 2>&1)
+        if echo "$result" | grep -q "OK: filestores match"; then
+            echo "$UUID  OK"
+        elif echo "$result" | grep -q "not running"; then
+            echo "$UUID  NOT RUNNING"
+        elif echo "$result" | grep -q "host filestore not found"; then
+            echo "$UUID  NO FILESTORE"
+        elif echo "$result" | grep -q "cannot query DB"; then
+            echo "$UUID  DB ERROR"
+        else
+            echo "$UUID  MISMATCH"
+        fi
+    done
+fi

app/sbin/rotateLogs

@@ -0,0 +1,43 @@
+#!/bin/bash
+# rotateLogs — rotate odoo.log for all UUID containers, keep 4 weeks
+# Runs as a daemon: waits for midnight, rotates, then waits for next midnight
+
+RETENTION_DAYS=28
+
+wait_for_midnight() {
+    ELAPSED=$(( $(date +%H) * 3600 + $(date +%M) * 60 + $(date +%S) ))
+    SLEEP=$(( 86400 - ELAPSED ))
+    echo "Waiting ${SLEEP}s until midnight..."
+    sleep "$SLEEP"
+}
+
+while true; do
+    wait_for_midnight
+
+for log in /4server/data/00*/logs/odoo.log; do
+    [[ -f "$log" ]] || continue
+    DIR=$(dirname "$log")
+    UUID=$(basename "$(dirname "$DIR")")
+
+    STAMP=$(date '+%Y-%m-%d_%H-%M')
+    ARCHIVE="$DIR/odoo.log.$STAMP"
+
+    # Copy preserving permissions, then truncate the live log
+    PERMS=$(stat -c '%a' "$log")
+    OWNER=$(stat -c '%u:%g' "$log")
+    cp "$log" "$ARCHIVE"
+    : > "$log"
+
+    # Compress the archive, restore original ownership and permissions
+    gzip "$ARCHIVE"
+    chmod "$PERMS" "${ARCHIVE}.gz"
+    chown "$OWNER" "${ARCHIVE}.gz"
+
+    echo "$UUID  rotated -> $(basename "${ARCHIVE}.gz")"
+
+    # Delete rotated logs older than 4 weeks
+    find "$DIR" -name 'odoo.log.*.gz' -mtime +$RETENTION_DAYS -exec rm -f {} \; \
+        -exec echo "$UUID  deleted: {}" \;
+done
+
+done

app/templates/init.d/rotateLogs

@@ -0,0 +1,22 @@
+#!/sbin/openrc-run
+
+name="rotateLogs"
+description="Daily Odoo log rotation"
+
+pidfile="/run/${RC_SVCNAME}.pid"
+output_log="/4server/data/log/rotateLogs.log"
+error_log="/4server/data/log/rotateLogs.log"
+
+command="/4server/sbin/rotateLogs"
+command_background="yes"
+
+stop() {
+    ebegin "Stopping rotateLogs"
+    start-stop-daemon --stop --pidfile "$pidfile"
+    eend $?
+}
+
+depend() {
+    need localmount
+    after bootmisc
+}

app/templates/postgresql.conf

@@ -0,0 +1,17 @@
+listen_addresses = '*'
+max_connections = 150
+dynamic_shared_memory_type = posix
+max_wal_size = 1GB
+min_wal_size = 80MB
+log_timezone = 'Etc/UTC'
+datestyle = 'iso, mdy'
+timezone = 'Etc/UTC'
+lc_messages = 'en_US.utf8'
+lc_monetary = 'en_US.utf8'
+lc_numeric = 'en_US.utf8'
+lc_time = 'en_US.utf8'
+default_text_search_config = 'pg_catalog.english'
+shared_buffers = 2GB
+effective_cache_size = 10GB
+effective_io_concurrency = 100
+random_page_cost = 2

app/update

@@ -68,6 +68,13 @@ rex doas chown root:root /etc/init.d/cleanTmp
 rex doas rc-update add cleanTmp default
 rex doas rc-service cleanTmp restart
 
+#INSTALL rotateLogs SERVICE
+template templates/init.d/rotateLogs /etc/init.d/rotateLogs
+rex doas chmod 0755 /etc/init.d/rotateLogs
+rex doas chown root:root /etc/init.d/rotateLogs
+rex doas rc-update add rotateLogs default
+rex doas rc-service rotateLogs restart
+
 #INSTALL cpu service
 template templates/init.d/cpu /etc/init.d/cpu
 rex doas chmod 0755 /etc/init.d/cpu

app/update_sbin

@@ -3,6 +3,7 @@
 echo "Running prsync ./sbin"
 prsync -h "/app/host_vars/hosts" -avz ./sbin/ /4server/sbin/
 template templates/.profile /home/4server/.profile
+template templates/postgresql.conf /4server/data/postgres/data/postgresql.conf
 
 rex doas rc-service api restart