From fbde9a19ab104f21bbeece36a3443001b0484d01 Mon Sep 17 00:00:00 2001 From: oliver Date: Fri, 14 Aug 2026 09:31:39 -0300 Subject: [PATCH] Create AGENTS.md --- AGENTS.md | 308 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 308 insertions(+) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..8c47717 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,308 @@ +# AGENTS.md — 4server Project Reference + +## What This Is + +**4server** is a self-hosted infrastructure management platform. It runs a Docker-based Alpine Linux control container locally and manages a fleet of remote Alpine Linux servers. Servers host multi-tenant Odoo (17/18/19) and N8N instances as Docker containers, all behind a Traefik reverse proxy with auto-TLS. + +--- + +## Repository Layout + +``` +4server/ +├── alpine/ # Control container (Dockerfile + entrypoint scripts) +│ ├── Dockerfile # Alpine image with pssh, borg, sshfs, nvim, cryptsetup, etc. +│ ├── config # SSH config for all managed hosts (user: 4server) +│ ├── known_hosts # Pre-populated SSH known_hosts +│ ├── rex # Run a command on ALL hosts via pssh +│ ├── template # Deploy a template file to ALL hosts (with variable substitution) +│ ├── create_volume # Create and LUKS-format an encrypted volume +│ └── mount_volume # Mount an encrypted volume +│ +├── app/ # Mounted as /app inside the control container +│ ├── host_vars/ # Per-host SSH keys, env files, and hosts list +│ │ ├── hosts # Plain-text list of hostnames for pssh/prsync +│ │ ├── create # Script to generate Ed25519 SSH key pairs +│ │ └── {host}/ # Directory per host: {host} (private key), {host}.pub, {host}.env +│ │ +│ ├── sbin/ # Scripts deployed to /4server/sbin/ on every server +│ │ ├── api # FastAPI REST API (Python 3, uvicorn) — server management API +│ │ ├── helpers # Bash lib: get_contract_info() — loads UUID data from SQLite +│ │ ├── startContainer # Dispatch start by UUID type +│ │ ├── stopContainer # Stop a container +│ │ ├── nukeContainer # Destroy container (must be stopped first) +│ │ ├── backupContainer # Dispatch backup by UUID type +│ │ ├── backupAll # Backup all running NNN-NNN containers +│ │ ├── borgpush # Push /BACKUP/current* files to BorgBase +│ │ ├── stats # JSON: uptime, top, cgroup, mounts +│ │ ├── audit # JSON: full Docker + OS audit +│ │ ├── checkCalls # OpenRC service: monitors API calls +│ │ ├── cleanTmp # OpenRC service: cleans /4server/tmp +│ │ ├── cpu # OpenRC service: CPU monitoring +│ │ ├── gitPull/gitRevert # Git operations inside containers +│ │ ├── migrate # Data migration helpers +│ │ ├── sql / psql # PostgreSQL query helpers +│ │ ├── contractInfo # Print contract info for a UUID +│ │ ├── getContainers # List containers +│ │ ├── pullAllContainers # docker pull on all containers +│ │ ├── restartContainers # Restart all containers +│ │ ├── odoo_passwd # Reset Odoo user password +│ │ ├── start/ # Type-specific start scripts (n8n, ODOO_17/18/19) +│ │ ├── backup/ # Type-specific backup scripts (N8N, ODOO) +│ │ ├── nuke/ # Type-specific nuke scripts (n8n, ODOO_19) +│ │ ├── ODOO_19/ # Odoo 19 admin tools (restore, shell, updateModules, audit, etc.) +│ │ └── N8N/ # N8N admin tools (currently empty) +│ │ +│ ├── templates/ # Config templates deployed via the `template` command +│ │ ├── docker-compose.yml # Base infra: postgres (beedb) + traefik +│ │ ├── traefik.yaml # Traefik config +│ │ ├── repositories # APK repository list +│ │ ├── hosts # /etc/hosts +│ │ ├── hostname # /etc/hostname +│ │ ├── extlinux.conf # Bootloader (variants for DO, Hostinger) +│ │ ├── nebula/ # Nebula VPN: config.yml, ca.crt, host.crt, host.key +│ │ ├── init.d/ # OpenRC service files: nebula, api, checkCalls, cleanTmp, cpu, ping_service +│ │ ├── ssh/ # SSH authorized_keys (id_ed25519.pub) +│ │ ├── etc/doas.d/ # doas config (4server.conf) +│ │ └── .profile # Shell profile for 4server user +│ │ +│ ├── etc/traefik/certs/ # TLS certificates distributed to servers +│ ├── vault/ # Encrypted secrets (not committed) +│ │ +│ ├── onboarding # New server bootstrap (Alpine packages, Nebula VPN, 4server user) +│ ├── hardening # Lock down SSH (no root, no password login) +│ ├── update # Full server update (packages, Docker infra, API, Traefik) +│ ├── update_sbin # Deploy /app/sbin/ to servers + restart services +│ ├── update_alpine # APK update/upgrade only +│ ├── restart # Restart core services +│ ├── backup_prune # Prune old Borg archives +│ ├── migrate # One-off DB migrations +│ └── download_sbin # Download binary tools (nebula, etc.) +│ +├── exchange/ # Shared volume between host and control container (file drop) +├── docker-compose.yaml # Builds + runs the Alpine control container +├── mount # Helper to mount exchange volume +└── start # Start the control container +``` + +--- + +## Servers (Managed Fleet) + +| Alias | IP | Port | Notes | +|------------|-----------------|------|--------------------| +| dev | 192.168.9.221 | 2222 | Development server | +| manchester | 192.168.9.20 | 22 | | +| boston | 192.168.9.16 | 22 | | +| mumbai | 192.168.9.17 | 22 | | +| meppel | 192.168.9.21 | 22 | | +| sydney | 192.168.9.22 | 22 | | +| saopaulo | 192.168.9.11 | 22 | | + +SSH user on all servers: `4server` +Key per host: `/app/host_vars/{host}/{host}` (Ed25519) +Active hosts file: `/app/host_vars/hosts` (currently: `sydney2`) + +--- + +## UUID Format + +Every managed container has a UUID structured as: `{server}-{type}-{random}` + +| Segment | Meaning | Known values | +|---------|-------------|-------------------------------------------| +| `aaa` | Server | `001` = manchester, `002` = boston | +| `bbb` | Image type | `001` = N8N, `002` = Odoo 18, `003` = Odoo 19, `004` = Odoo 17 | + +The second segment (`bbb`) is used by `startContainer`, `nukeContainer`, `backupContainer` to dispatch to the right type-specific script. + +--- + +## Core Tools (inside the Alpine control container) + +### `rex ` +Runs a shell command on **all hosts** in `/app/host_vars/hosts` via `pssh`. +Example: `rex doas apk update` + +### `prsync -h /app/host_vars/hosts -avz ` +Parallel rsync to all hosts. Used to deploy `/app/sbin/` and certs. + +### `template ` +Renders `{{KEY}}` placeholders in `localfile` and copies it to `remotefile` on every host. +Available template variables (loaded from `host_vars/{host}/{host}.env` + key files): +- `NEBULA_CA`, `NEBULA_CRT`, `NEBULA_KEY` +- `API_KEY`, `HOSTNAME` +- `SSH_PRIVATE`, `SSH_PUBLIC` + +--- + +## Server-Side Architecture (on each managed server) + +``` +/4server/ +├── sbin/ # All management scripts (deployed from app/sbin/) +├── data/ +│ ├── contracts.db # SQLite: container contracts table +│ ├── log/ # Logs for all services +│ ├── {UUID}/ # Per-container data directory +│ │ ├── odoo/ # Odoo filestore +│ │ ├── git/ # Git repos +│ │ ├── git-server/ # Bare git repos exposed via SSH +│ │ ├── logs/ +│ │ ├── config/ +│ │ ├── cc/ # Control channel (e.g. backup trigger file) +│ │ ├── etc/ # domain file, gitpath, etc. +│ │ ├── install/ +│ │ └── .ssh/ # Container SSH keys +│ ├── traefik/etc/ # traefik.yaml + TLS certs +│ └── postgres/ # PostgreSQL data + pg_hba.conf +├── tmp/ # Temporary files (cleaned by cleanTmp service) +└── docker-compose.yml # Base infra (beedb + traefik) + +/BACKUP/{UUID}/ # Per-container backup files (borgpush picks up current*) +``` + +### Persistent Docker Services (base infra) +| Container | Image | Role | IP (internal) | +|-----------|---------------|-------------------------------|----------------| +| `beedb` | postgres:16 | Shared PostgreSQL for all Odoo| 10.5.0.200 | +| `traefik` | traefik:3.6 | Reverse proxy, HTTPS (Let's Encrypt) | — | + +Docker network: `4server_4projects` (bridge, `10.5.0.0/16`) + +### Per-Container Docker Services +- Each UUID is a `docker run` container on the host's Docker network +- SSH port: `CONTAINERDBID + 2200` (forwarded to the host) +- Traefik routing via Docker labels (`traefik.http.routers.{UUID}.*`) +- Default domain: `{UUID}.odoo4projects.com`; optional custom domains from `/4server/data/{UUID}/etc/domain` + +--- + +## Data Model: `contracts.db` + +Table: `containers` + +| Column | Description | +|----------------|----------------------------------------------------| +| `UUID` | Container identifier (aaa-bbb-... format) | +| `id` | Auto-increment row ID (`CONTAINERDBID`) | +| `image` | Image type code | +| `status` | Container status | +| `expires` | Contract expiry date | +| `created` | Creation timestamp | +| `secret_list` | Backslash-separated `key=value` pairs (passwords) | +| `git` | Git integration config | +| `backup_slots` | Number of backup slots | +| `hdd` | Disk quota | +| `workers` | Worker count | +| `domains_slots`| Number of allowed custom domains | +| `domains` | Active custom domains (comma-separated) | +| `affiliate` | JSON: UTM tracking data | + +`secret_list` parsing: `key=value` pairs separated by `\`. Key `psql` = Odoo DB password (`ODOO_DB_PASSWORD`). + +--- + +## Management API (`/4server/sbin/api`) + +FastAPI Python 3 app, served by uvicorn, managed by OpenRC (`api` service). + +- Auth: `X-API-Key` header (value from `/etc/4server` env file, key `API_KEY`) +- Version: `0.0.9` + +Key endpoints (all protected): +| Endpoint | Description | +|----------|-------------| +| `GET /containers` | List all containers from SQLite | +| `POST /container/update` | Create/update a container record | +| `POST /container/start` | Start a container by UUID | +| `POST /container/stop` | Stop a container by UUID | +| `POST /container/nuke` | Destroy a container | +| `GET /container/info` | Full container info + Docker inspect | +| `POST /container/bump` | Bump/restart a container | +| `GET /container/quota` | Disk quota usage | +| `GET /images` | List available Docker images | +| `GET /cpu` | CPU log | +| `GET /stats` | Server runtime stats (JSON) | +| `GET /audit` | Full Docker + OS audit (JSON) | +| `POST /git` | Git pull/revert inside container | +| `POST /git/key` | Add SSH key to git-server | +| `GET /odoo/audit` | Odoo-specific audit | +| `GET /odoo/log` | Summarised Odoo log (last N lines, errors/warnings) | +| `POST /backup/import` | Import a backup archive | +| `POST /backup/move` | Move a backup file | +| `POST /backup/check` | Check backup integrity | +| `POST /backup/all` | Trigger backup of all containers | +| `POST /backup/borgpush` | Push backups to BorgBase | + +--- + +## Backup + +- **Per-container**: `backupContainer ` → dispatches to `backup/N8N` or `backup/ODOO` +- **All containers**: `backupAll` — touches `/4server/data/{UUID}/cc/backup` for each running NNN-NNN container +- **Off-site**: `borgpush` → BorgBase (`e7e9h45y.repo.borgbase.com`) via SSH key `~/.ssh/borg-backup` +- **Pruning**: `backup_prune` — keep 7 daily / 8 weekly / 12 monthly + +--- + +## Security Model + +- **Privilege escalation**: `doas` (not sudo). Config at `/etc/doas.d/4server.conf` +- **SSH**: Key-only (`PasswordAuthentication no`, `PermitRootLogin no`) +- **API**: Header-based API key (`X-API-Key`) +- **Secrets**: Stored in `secret_list` column of SQLite (base64-safe `\`-separated pairs) +- **VPN**: Nebula overlay mesh between all servers (certs per host in `host_vars/{host}/`) +- **TLS**: Traefik with Let's Encrypt production resolver + +--- + +## Workflow Reference + +### Add a new server +1. Add SSH key pair: `cd app/host_vars && ./create {hostname}` +2. Create `host_vars/{host}/{host}.env` with `HOSTNAME`, `API_KEY`, `NEBULA_CRT`, `NEBULA_KEY` +3. Add host to `host_vars/hosts` +4. Add SSH config block to `alpine/config` +5. Run `onboarding` from inside the control container +6. Run `hardening` to lock down SSH + +### Deploy code changes to servers +```bash +# Inside control container (/app) +./update_sbin # deploy sbin/ scripts + restart api/checkCalls +``` + +### Deploy a config template +```bash +template templates/my.conf /etc/my.conf +``` + +### Run a command on all servers +```bash +rex doas rc-service api restart +``` + +### Start/stop a specific container (from server) +```bash +startContainer +stopContainer +``` + +--- + +## Key Technologies + +| Layer | Technology | +|-------|------------| +| Control container | Alpine Linux + Docker Compose | +| Server OS | Alpine Linux (OpenRC init system) | +| Reverse proxy | Traefik v3.6 | +| Database | PostgreSQL 16 (shared, per-server) | +| App runtime | Odoo 17/18/19, N8N | +| Management API | Python 3 / FastAPI / uvicorn | +| Backup | BorgBackup → BorgBase | +| VPN | Nebula mesh | +| Parallel SSH | pssh / prsync | +| Container secrets | SQLite (`contracts.db`) | +| SSH auth | Ed25519 keys per host |