4server/AGENTS.md

AGENTS.md — 4server Project Reference

What This Is

4server is a self-hosted infrastructure management platform. It runs a Docker-based Alpine Linux control container locally and manages a fleet of remote Alpine Linux servers. Servers host multi-tenant Odoo (17/18/19) and N8N instances as Docker containers, all behind a Traefik reverse proxy with auto-TLS.

---

Repository Layout

4server/
├── alpine/             # Control container (Dockerfile + entrypoint scripts)
│   ├── Dockerfile      # Alpine image with pssh, borg, sshfs, nvim, cryptsetup, etc.
│   ├── config          # SSH config for all managed hosts (user: 4server)
│   ├── known_hosts     # Pre-populated SSH known_hosts
│   ├── rex             # Run a command on ALL hosts via pssh
│   ├── template        # Deploy a template file to ALL hosts (with variable substitution)
│   ├── create_volume   # Create and LUKS-format an encrypted volume
│   └── mount_volume    # Mount an encrypted volume
│
├── app/                # Mounted as /app inside the control container
│   ├── host_vars/      # Per-host SSH keys, env files, and hosts list
│   │   ├── hosts       # Plain-text list of hostnames for pssh/prsync
│   │   ├── create      # Script to generate Ed25519 SSH key pairs
│   │   └── {host}/     # Directory per host: {host} (private key), {host}.pub, {host}.env
│   │
│   ├── sbin/           # Scripts deployed to /4server/sbin/ on every server
│   │   ├── api         # FastAPI REST API (Python 3, uvicorn) — server management API
│   │   ├── helpers     # Bash lib: get_contract_info() — loads UUID data from SQLite
│   │   ├── startContainer    # Dispatch start by UUID type
│   │   ├── stopContainer     # Stop a container
│   │   ├── nukeContainer     # Destroy container (must be stopped first)
│   │   ├── backupContainer   # Dispatch backup by UUID type
│   │   ├── backupAll         # Backup all running NNN-NNN containers
│   │   ├── borgpush          # Push /BACKUP/current* files to BorgBase
│   │   ├── stats             # JSON: uptime, top, cgroup, mounts
│   │   ├── audit             # JSON: full Docker + OS audit
│   │   ├── checkCalls        # OpenRC service: monitors API calls
│   │   ├── cleanTmp          # OpenRC service: cleans /4server/tmp
│   │   ├── cpu               # OpenRC service: CPU monitoring
│   │   ├── gitPull/gitRevert # Git operations inside containers
│   │   ├── migrate           # Data migration helpers
│   │   ├── sql / psql        # PostgreSQL query helpers
│   │   ├── contractInfo      # Print contract info for a UUID
│   │   ├── getContainers     # List containers
│   │   ├── pullAllContainers # docker pull on all containers
│   │   ├── restartContainers # Restart all containers
│   │   ├── odoo_passwd       # Reset Odoo user password
│   │   ├── start/            # Type-specific start scripts (n8n, ODOO_17/18/19)
│   │   ├── backup/           # Type-specific backup scripts (N8N, ODOO)
│   │   ├── nuke/             # Type-specific nuke scripts (n8n, ODOO_19)
│   │   ├── ODOO_19/          # Odoo 19 admin tools (restore, shell, updateModules, audit, etc.)
│   │   └── N8N/              # N8N admin tools (currently empty)
│   │
│   ├── templates/      # Config templates deployed via the `template` command
│   │   ├── docker-compose.yml   # Base infra: postgres (beedb) + traefik
│   │   ├── traefik.yaml         # Traefik config
│   │   ├── repositories         # APK repository list
│   │   ├── hosts                # /etc/hosts
│   │   ├── hostname             # /etc/hostname
│   │   ├── extlinux.conf        # Bootloader (variants for DO, Hostinger)
│   │   ├── nebula/              # Nebula VPN: config.yml, ca.crt, host.crt, host.key
│   │   ├── init.d/              # OpenRC service files: nebula, api, checkCalls, cleanTmp, cpu, ping_service
│   │   ├── ssh/                 # SSH authorized_keys (id_ed25519.pub)
│   │   ├── etc/doas.d/          # doas config (4server.conf)
│   │   └── .profile             # Shell profile for 4server user
│   │
│   ├── etc/traefik/certs/  # TLS certificates distributed to servers
│   ├── vault/              # Encrypted secrets (not committed)
│   │
│   ├── onboarding      # New server bootstrap (Alpine packages, Nebula VPN, 4server user)
│   ├── hardening       # Lock down SSH (no root, no password login)
│   ├── update          # Full server update (packages, Docker infra, API, Traefik)
│   ├── update_sbin     # Deploy /app/sbin/ to servers + restart services
│   ├── update_alpine   # APK update/upgrade only
│   ├── restart         # Restart core services
│   ├── backup_prune    # Prune old Borg archives
│   ├── migrate         # One-off DB migrations
│   └── download_sbin   # Download binary tools (nebula, etc.)
│
├── exchange/           # Shared volume between host and control container (file drop)
├── docker-compose.yaml # Builds + runs the Alpine control container
├── mount               # Helper to mount exchange volume
└── start               # Start the control container

---

Servers (Managed Fleet)

Alias	IP	Port	Notes
dev	192.168.9.221	2222	Development server
manchester	192.168.9.20	22
boston	192.168.9.16	22
mumbai	192.168.9.17	22
meppel	192.168.9.21	22
sydney	192.168.9.22	22
saopaulo	192.168.9.11	22

SSH user on all servers: 4server
Key per host: /app/host_vars/{host}/{host} (Ed25519)
Active hosts file: /app/host_vars/hosts (currently: sydney2)

---

UUID Format

Every managed container has a UUID structured as: {server}-{type}-{random}

Segment	Meaning	Known values
aaa	Server	001 = manchester, 002 = boston
bbb	Image type	001 = N8N, 002 = Odoo 18, 003 = Odoo 19, 004 = Odoo 17

The second segment (bbb) is used by startContainer, nukeContainer, backupContainer to dispatch to the right type-specific script.

---

Core Tools (inside the Alpine control container)

rex <command>

Runs a shell command on all hosts in /app/host_vars/hosts via pssh.
Example: rex doas apk update

prsync -h /app/host_vars/hosts -avz <src> <dst>

Parallel rsync to all hosts. Used to deploy /app/sbin/ and certs.

template <localfile> <remotefile>

Renders {{KEY}} placeholders in localfile and copies it to remotefile on every host.
Available template variables (loaded from host_vars/{host}/{host}.env + key files):

• NEBULA_CA, NEBULA_CRT, NEBULA_KEY
• API_KEY, HOSTNAME
• SSH_PRIVATE, SSH_PUBLIC

---

Server-Side Architecture (on each managed server)

/4server/
├── sbin/               # All management scripts (deployed from app/sbin/)
├── data/
│   ├── contracts.db    # SQLite: container contracts table
│   ├── log/            # Logs for all services
│   ├── {UUID}/         # Per-container data directory
│   │   ├── odoo/       # Odoo filestore
│   │   ├── git/        # Git repos
│   │   ├── git-server/ # Bare git repos exposed via SSH
│   │   ├── logs/
│   │   ├── config/
│   │   ├── cc/         # Control channel (e.g. backup trigger file)
│   │   ├── etc/        # domain file, gitpath, etc.
│   │   ├── install/
│   │   └── .ssh/       # Container SSH keys
│   ├── traefik/etc/    # traefik.yaml + TLS certs
│   └── postgres/       # PostgreSQL data + pg_hba.conf
├── tmp/                # Temporary files (cleaned by cleanTmp service)
└── docker-compose.yml  # Base infra (beedb + traefik)

/BACKUP/{UUID}/         # Per-container backup files (borgpush picks up current*)

Persistent Docker Services (base infra)

Container	Image	Role	IP (internal)
beedb	postgres:16	Shared PostgreSQL for all Odoo	10.5.0.200
traefik	traefik:3.6	Reverse proxy, HTTPS (Let's Encrypt)	—

Docker network: 4server_4projects (bridge, 10.5.0.0/16)

Per-Container Docker Services

• Each UUID is a docker run container on the host's Docker network
• SSH port: CONTAINERDBID + 2200 (forwarded to the host)
• Traefik routing via Docker labels (traefik.http.routers.{UUID}.*)
• Default domain: {UUID}.odoo4projects.com; optional custom domains from /4server/data/{UUID}/etc/domain

---

Data Model: contracts.db

Table: containers

Column	Description
UUID	Container identifier (aaa-bbb-... format)
id	Auto-increment row ID (CONTAINERDBID)
image	Image type code
status	Container status
expires	Contract expiry date
created	Creation timestamp
secret_list	Backslash-separated key=value pairs (passwords)
git	Git integration config
backup_slots	Number of backup slots
hdd	Disk quota
workers	Worker count
domains_slots	Number of allowed custom domains
domains	Active custom domains (comma-separated)
affiliate	JSON: UTM tracking data

secret_list parsing: key=value pairs separated by \. Key psql = Odoo DB password (ODOO_DB_PASSWORD).

---

Management API (/4server/sbin/api)

FastAPI Python 3 app, served by uvicorn, managed by OpenRC (api service).

• Auth: X-API-Key header (value from /etc/4server env file, key API_KEY)
• Version: 0.0.9

Key endpoints (all protected):

Endpoint	Description
GET /containers	List all containers from SQLite
POST /container/update	Create/update a container record
POST /container/start	Start a container by UUID
POST /container/stop	Stop a container by UUID
POST /container/nuke	Destroy a container
GET /container/info	Full container info + Docker inspect
POST /container/bump	Bump/restart a container
GET /container/quota	Disk quota usage
GET /images	List available Docker images
GET /cpu	CPU log
GET /stats	Server runtime stats (JSON)
GET /audit	Full Docker + OS audit (JSON)
POST /git	Git pull/revert inside container
POST /git/key	Add SSH key to git-server
GET /odoo/audit	Odoo-specific audit
GET /odoo/log	Summarised Odoo log (last N lines, errors/warnings)
POST /backup/import	Import a backup archive
POST /backup/move	Move a backup file
POST /backup/check	Check backup integrity
POST /backup/all	Trigger backup of all containers
POST /backup/borgpush	Push backups to BorgBase

---

Backup

• Per-container: backupContainer <UUID> → dispatches to backup/N8N or backup/ODOO
• All containers: backupAll — touches /4server/data/{UUID}/cc/backup for each running NNN-NNN container
• Off-site: borgpush → BorgBase (e7e9h45y.repo.borgbase.com) via SSH key ~/.ssh/borg-backup
• Pruning: backup_prune — keep 7 daily / 8 weekly / 12 monthly

---

Security Model

• Privilege escalation: doas (not sudo). Config at /etc/doas.d/4server.conf
• SSH: Key-only (PasswordAuthentication no, PermitRootLogin no)
• API: Header-based API key (X-API-Key)
• Secrets: Stored in secret_list column of SQLite (base64-safe \-separated pairs)
• VPN: Nebula overlay mesh between all servers (certs per host in host_vars/{host}/)
• TLS: Traefik with Let's Encrypt production resolver

---

Workflow Reference

Add a new server

1. Add SSH key pair: cd app/host_vars && ./create {hostname}
2. Create host_vars/{host}/{host}.env with HOSTNAME, API_KEY, NEBULA_CRT, NEBULA_KEY
3. Add host to host_vars/hosts
4. Add SSH config block to alpine/config
5. Run onboarding from inside the control container
6. Run hardening to lock down SSH

Deploy code changes to servers

# Inside control container (/app)
./update_sbin        # deploy sbin/ scripts + restart api/checkCalls

Deploy a config template

template templates/my.conf /etc/my.conf

Run a command on all servers

rex doas rc-service api restart

Start/stop a specific container (from server)

startContainer <UUID>
stopContainer <UUID>

---

Key Technologies

Layer	Technology
Control container	Alpine Linux + Docker Compose
Server OS	Alpine Linux (OpenRC init system)
Reverse proxy	Traefik v3.6
Database	PostgreSQL 16 (shared, per-server)
App runtime	Odoo 17/18/19, N8N
Management API	Python 3 / FastAPI / uvicorn
Backup	BorgBackup → BorgBase
VPN	Nebula mesh
Parallel SSH	pssh / prsync
Container secrets	SQLite (contracts.db)
SSH auth	Ed25519 keys per host