# AGENTS.md — 4server Project Reference ## What This Is **4server** is a self-hosted infrastructure management platform. It runs a Docker-based Alpine Linux control container locally and manages a fleet of remote Alpine Linux servers. Servers host multi-tenant Odoo (17/18/19) and N8N instances as Docker containers, all behind a Traefik reverse proxy with auto-TLS. --- ## Repository Layout ``` 4server/ ├── alpine/ # Control container (Dockerfile + entrypoint scripts) │ ├── Dockerfile # Alpine image with pssh, borg, sshfs, nvim, cryptsetup, etc. │ ├── config # SSH config for all managed hosts (user: 4server) │ ├── known_hosts # Pre-populated SSH known_hosts │ ├── rex # Run a command on ALL hosts via pssh │ ├── template # Deploy a template file to ALL hosts (with variable substitution) │ ├── create_volume # Create and LUKS-format an encrypted volume │ └── mount_volume # Mount an encrypted volume │ ├── app/ # Mounted as /app inside the control container │ ├── host_vars/ # Per-host SSH keys, env files, and hosts list │ │ ├── hosts # Plain-text list of hostnames for pssh/prsync │ │ ├── create # Script to generate Ed25519 SSH key pairs │ │ └── {host}/ # Directory per host: {host} (private key), {host}.pub, {host}.env │ │ │ ├── sbin/ # Scripts deployed to /4server/sbin/ on every server │ │ ├── api # FastAPI REST API (Python 3, uvicorn) — server management API │ │ ├── helpers # Bash lib: get_contract_info() — loads UUID data from SQLite │ │ ├── startContainer # Dispatch start by UUID type │ │ ├── stopContainer # Stop a container │ │ ├── nukeContainer # Destroy container (must be stopped first) │ │ ├── backupContainer # Dispatch backup by UUID type │ │ ├── backupAll # Backup all running NNN-NNN containers │ │ ├── borgpush # Push /BACKUP/current* files to BorgBase │ │ ├── stats # JSON: uptime, top, cgroup, mounts │ │ ├── audit # JSON: full Docker + OS audit │ │ ├── checkCalls # OpenRC service: monitors API calls │ │ ├── cleanTmp # OpenRC service: cleans /4server/tmp │ │ ├── cpu # OpenRC service: CPU monitoring │ │ ├── gitPull/gitRevert # Git operations inside containers │ │ ├── migrate # Data migration helpers │ │ ├── sql / psql # PostgreSQL query helpers │ │ ├── contractInfo # Print contract info for a UUID │ │ ├── getContainers # List containers │ │ ├── pullAllContainers # docker pull on all containers │ │ ├── restartContainers # Restart all containers │ │ ├── odoo_passwd # Reset Odoo user password │ │ ├── start/ # Type-specific start scripts (n8n, ODOO_17/18/19) │ │ ├── backup/ # Type-specific backup scripts (N8N, ODOO) │ │ ├── nuke/ # Type-specific nuke scripts (n8n, ODOO_19) │ │ ├── ODOO_19/ # Odoo 19 admin tools (restore, shell, updateModules, audit, etc.) │ │ └── N8N/ # N8N admin tools (currently empty) │ │ │ ├── templates/ # Config templates deployed via the `template` command │ │ ├── docker-compose.yml # Base infra: postgres (beedb) + traefik │ │ ├── traefik.yaml # Traefik config │ │ ├── repositories # APK repository list │ │ ├── hosts # /etc/hosts │ │ ├── hostname # /etc/hostname │ │ ├── extlinux.conf # Bootloader (variants for DO, Hostinger) │ │ ├── nebula/ # Nebula VPN: config.yml, ca.crt, host.crt, host.key │ │ ├── init.d/ # OpenRC service files: nebula, api, checkCalls, cleanTmp, cpu, ping_service │ │ ├── ssh/ # SSH authorized_keys (id_ed25519.pub) │ │ ├── etc/doas.d/ # doas config (4server.conf) │ │ └── .profile # Shell profile for 4server user │ │ │ ├── etc/traefik/certs/ # TLS certificates distributed to servers │ ├── vault/ # Encrypted secrets (not committed) │ │ │ ├── onboarding # New server bootstrap (Alpine packages, Nebula VPN, 4server user) │ ├── hardening # Lock down SSH (no root, no password login) │ ├── update # Full server update (packages, Docker infra, API, Traefik) │ ├── update_sbin # Deploy /app/sbin/ to servers + restart services │ ├── update_alpine # APK update/upgrade only │ ├── restart # Restart core services │ ├── backup_prune # Prune old Borg archives │ ├── migrate # One-off DB migrations │ └── download_sbin # Download binary tools (nebula, etc.) │ ├── exchange/ # Shared volume between host and control container (file drop) ├── docker-compose.yaml # Builds + runs the Alpine control container ├── mount # Helper to mount exchange volume └── start # Start the control container ``` --- ## Servers (Managed Fleet) | Alias | IP | Port | Notes | |------------|-----------------|------|--------------------| | dev | 192.168.9.221 | 2222 | Development server | | manchester | 192.168.9.20 | 22 | | | boston | 192.168.9.16 | 22 | | | mumbai | 192.168.9.17 | 22 | | | meppel | 192.168.9.21 | 22 | | | sydney | 192.168.9.22 | 22 | | | saopaulo | 192.168.9.11 | 22 | | SSH user on all servers: `4server` Key per host: `/app/host_vars/{host}/{host}` (Ed25519) Active hosts file: `/app/host_vars/hosts` (currently: `sydney2`) --- ## UUID Format Every managed container has a UUID structured as: `{server}-{type}-{random}` | Segment | Meaning | Known values | |---------|-------------|-------------------------------------------| | `aaa` | Server | `001` = manchester, `002` = boston | | `bbb` | Image type | `001` = N8N, `002` = Odoo 18, `003` = Odoo 19, `004` = Odoo 17 | The second segment (`bbb`) is used by `startContainer`, `nukeContainer`, `backupContainer` to dispatch to the right type-specific script. --- ## Core Tools (inside the Alpine control container) ### `rex ` Runs a shell command on **all hosts** in `/app/host_vars/hosts` via `pssh`. Example: `rex doas apk update` ### `prsync -h /app/host_vars/hosts -avz ` Parallel rsync to all hosts. Used to deploy `/app/sbin/` and certs. ### `template ` Renders `{{KEY}}` placeholders in `localfile` and copies it to `remotefile` on every host. Available template variables (loaded from `host_vars/{host}/{host}.env` + key files): - `NEBULA_CA`, `NEBULA_CRT`, `NEBULA_KEY` - `API_KEY`, `HOSTNAME` - `SSH_PRIVATE`, `SSH_PUBLIC` --- ## Server-Side Architecture (on each managed server) ``` /4server/ ├── sbin/ # All management scripts (deployed from app/sbin/) ├── data/ │ ├── contracts.db # SQLite: container contracts table │ ├── log/ # Logs for all services │ ├── {UUID}/ # Per-container data directory │ │ ├── odoo/ # Odoo filestore │ │ ├── git/ # Git repos │ │ ├── git-server/ # Bare git repos exposed via SSH │ │ ├── logs/ │ │ ├── config/ │ │ ├── cc/ # Control channel (e.g. backup trigger file) │ │ ├── etc/ # domain file, gitpath, etc. │ │ ├── install/ │ │ └── .ssh/ # Container SSH keys │ ├── traefik/etc/ # traefik.yaml + TLS certs │ └── postgres/ # PostgreSQL data + pg_hba.conf ├── tmp/ # Temporary files (cleaned by cleanTmp service) └── docker-compose.yml # Base infra (beedb + traefik) /BACKUP/{UUID}/ # Per-container backup files (borgpush picks up current*) ``` ### Persistent Docker Services (base infra) | Container | Image | Role | IP (internal) | |-----------|---------------|-------------------------------|----------------| | `beedb` | postgres:16 | Shared PostgreSQL for all Odoo| 10.5.0.200 | | `traefik` | traefik:3.6 | Reverse proxy, HTTPS (Let's Encrypt) | — | Docker network: `4server_4projects` (bridge, `10.5.0.0/16`) ### Per-Container Docker Services - Each UUID is a `docker run` container on the host's Docker network - SSH port: `CONTAINERDBID + 2200` (forwarded to the host) - Traefik routing via Docker labels (`traefik.http.routers.{UUID}.*`) - Default domain: `{UUID}.odoo4projects.com`; optional custom domains from `/4server/data/{UUID}/etc/domain` --- ## Data Model: `contracts.db` Table: `containers` | Column | Description | |----------------|----------------------------------------------------| | `UUID` | Container identifier (aaa-bbb-... format) | | `id` | Auto-increment row ID (`CONTAINERDBID`) | | `image` | Image type code | | `status` | Container status | | `expires` | Contract expiry date | | `created` | Creation timestamp | | `secret_list` | Backslash-separated `key=value` pairs (passwords) | | `git` | Git integration config | | `backup_slots` | Number of backup slots | | `hdd` | Disk quota | | `workers` | Worker count | | `domains_slots`| Number of allowed custom domains | | `domains` | Active custom domains (comma-separated) | | `affiliate` | JSON: UTM tracking data | `secret_list` parsing: `key=value` pairs separated by `\`. Key `psql` = Odoo DB password (`ODOO_DB_PASSWORD`). --- ## Management API (`/4server/sbin/api`) FastAPI Python 3 app, served by uvicorn, managed by OpenRC (`api` service). - Auth: `X-API-Key` header (value from `/etc/4server` env file, key `API_KEY`) - Version: `0.0.9` Key endpoints (all protected): | Endpoint | Description | |----------|-------------| | `GET /containers` | List all containers from SQLite | | `POST /container/update` | Create/update a container record | | `POST /container/start` | Start a container by UUID | | `POST /container/stop` | Stop a container by UUID | | `POST /container/nuke` | Destroy a container | | `GET /container/info` | Full container info + Docker inspect | | `POST /container/bump` | Bump/restart a container | | `GET /container/quota` | Disk quota usage | | `GET /images` | List available Docker images | | `GET /cpu` | CPU log | | `GET /stats` | Server runtime stats (JSON) | | `GET /audit` | Full Docker + OS audit (JSON) | | `POST /git` | Git pull/revert inside container | | `POST /git/key` | Add SSH key to git-server | | `GET /odoo/audit` | Odoo-specific audit | | `GET /odoo/log` | Summarised Odoo log (last N lines, errors/warnings) | | `POST /backup/import` | Import a backup archive | | `POST /backup/move` | Move a backup file | | `POST /backup/check` | Check backup integrity | | `POST /backup/all` | Trigger backup of all containers | | `POST /backup/borgpush` | Push backups to BorgBase | --- ## Backup - **Per-container**: `backupContainer ` → dispatches to `backup/N8N` or `backup/ODOO` - **All containers**: `backupAll` — touches `/4server/data/{UUID}/cc/backup` for each running NNN-NNN container - **Off-site**: `borgpush` → BorgBase (`e7e9h45y.repo.borgbase.com`) via SSH key `~/.ssh/borg-backup` - **Pruning**: `backup_prune` — keep 7 daily / 8 weekly / 12 monthly --- ## Security Model - **Privilege escalation**: `doas` (not sudo). Config at `/etc/doas.d/4server.conf` - **SSH**: Key-only (`PasswordAuthentication no`, `PermitRootLogin no`) - **API**: Header-based API key (`X-API-Key`) - **Secrets**: Stored in `secret_list` column of SQLite (base64-safe `\`-separated pairs) - **VPN**: Nebula overlay mesh between all servers (certs per host in `host_vars/{host}/`) - **TLS**: Traefik with Let's Encrypt production resolver --- ## Workflow Reference ### Add a new server 1. Add SSH key pair: `cd app/host_vars && ./create {hostname}` 2. Create `host_vars/{host}/{host}.env` with `HOSTNAME`, `API_KEY`, `NEBULA_CRT`, `NEBULA_KEY` 3. Add host to `host_vars/hosts` 4. Add SSH config block to `alpine/config` 5. Run `onboarding` from inside the control container 6. Run `hardening` to lock down SSH ### Deploy code changes to servers ```bash # Inside control container (/app) ./update_sbin # deploy sbin/ scripts + restart api/checkCalls ``` ### Deploy a config template ```bash template templates/my.conf /etc/my.conf ``` ### Run a command on all servers ```bash rex doas rc-service api restart ``` ### Start/stop a specific container (from server) ```bash startContainer stopContainer ``` --- ## Key Technologies | Layer | Technology | |-------|------------| | Control container | Alpine Linux + Docker Compose | | Server OS | Alpine Linux (OpenRC init system) | | Reverse proxy | Traefik v3.6 | | Database | PostgreSQL 16 (shared, per-server) | | App runtime | Odoo 17/18/19, N8N | | Management API | Python 3 / FastAPI / uvicorn | | Backup | BorgBackup → BorgBase | | VPN | Nebula mesh | | Parallel SSH | pssh / prsync | | Container secrets | SQLite (`contracts.db`) | | SSH auth | Ed25519 keys per host |