309 lines
14 KiB
Markdown
309 lines
14 KiB
Markdown
# AGENTS.md — 4server Project Reference
|
|
|
|
## What This Is
|
|
|
|
**4server** is a self-hosted infrastructure management platform. It runs a Docker-based Alpine Linux control container locally and manages a fleet of remote Alpine Linux servers. Servers host multi-tenant Odoo (17/18/19) and N8N instances as Docker containers, all behind a Traefik reverse proxy with auto-TLS.
|
|
|
|
---
|
|
|
|
## Repository Layout
|
|
|
|
```
|
|
4server/
|
|
├── alpine/ # Control container (Dockerfile + entrypoint scripts)
|
|
│ ├── Dockerfile # Alpine image with pssh, borg, sshfs, nvim, cryptsetup, etc.
|
|
│ ├── config # SSH config for all managed hosts (user: 4server)
|
|
│ ├── known_hosts # Pre-populated SSH known_hosts
|
|
│ ├── rex # Run a command on ALL hosts via pssh
|
|
│ ├── template # Deploy a template file to ALL hosts (with variable substitution)
|
|
│ ├── create_volume # Create and LUKS-format an encrypted volume
|
|
│ └── mount_volume # Mount an encrypted volume
|
|
│
|
|
├── app/ # Mounted as /app inside the control container
|
|
│ ├── host_vars/ # Per-host SSH keys, env files, and hosts list
|
|
│ │ ├── hosts # Plain-text list of hostnames for pssh/prsync
|
|
│ │ ├── create # Script to generate Ed25519 SSH key pairs
|
|
│ │ └── {host}/ # Directory per host: {host} (private key), {host}.pub, {host}.env
|
|
│ │
|
|
│ ├── sbin/ # Scripts deployed to /4server/sbin/ on every server
|
|
│ │ ├── api # FastAPI REST API (Python 3, uvicorn) — server management API
|
|
│ │ ├── helpers # Bash lib: get_contract_info() — loads UUID data from SQLite
|
|
│ │ ├── startContainer # Dispatch start by UUID type
|
|
│ │ ├── stopContainer # Stop a container
|
|
│ │ ├── nukeContainer # Destroy container (must be stopped first)
|
|
│ │ ├── backupContainer # Dispatch backup by UUID type
|
|
│ │ ├── backupAll # Backup all running NNN-NNN containers
|
|
│ │ ├── borgpush # Push /BACKUP/current* files to BorgBase
|
|
│ │ ├── stats # JSON: uptime, top, cgroup, mounts
|
|
│ │ ├── audit # JSON: full Docker + OS audit
|
|
│ │ ├── checkCalls # OpenRC service: monitors API calls
|
|
│ │ ├── cleanTmp # OpenRC service: cleans /4server/tmp
|
|
│ │ ├── cpu # OpenRC service: CPU monitoring
|
|
│ │ ├── gitPull/gitRevert # Git operations inside containers
|
|
│ │ ├── migrate # Data migration helpers
|
|
│ │ ├── sql / psql # PostgreSQL query helpers
|
|
│ │ ├── contractInfo # Print contract info for a UUID
|
|
│ │ ├── getContainers # List containers
|
|
│ │ ├── pullAllContainers # docker pull on all containers
|
|
│ │ ├── restartContainers # Restart all containers
|
|
│ │ ├── odoo_passwd # Reset Odoo user password
|
|
│ │ ├── start/ # Type-specific start scripts (n8n, ODOO_17/18/19)
|
|
│ │ ├── backup/ # Type-specific backup scripts (N8N, ODOO)
|
|
│ │ ├── nuke/ # Type-specific nuke scripts (n8n, ODOO_19)
|
|
│ │ ├── ODOO_19/ # Odoo 19 admin tools (restore, shell, updateModules, audit, etc.)
|
|
│ │ └── N8N/ # N8N admin tools (currently empty)
|
|
│ │
|
|
│ ├── templates/ # Config templates deployed via the `template` command
|
|
│ │ ├── docker-compose.yml # Base infra: postgres (beedb) + traefik
|
|
│ │ ├── traefik.yaml # Traefik config
|
|
│ │ ├── repositories # APK repository list
|
|
│ │ ├── hosts # /etc/hosts
|
|
│ │ ├── hostname # /etc/hostname
|
|
│ │ ├── extlinux.conf # Bootloader (variants for DO, Hostinger)
|
|
│ │ ├── nebula/ # Nebula VPN: config.yml, ca.crt, host.crt, host.key
|
|
│ │ ├── init.d/ # OpenRC service files: nebula, api, checkCalls, cleanTmp, cpu, ping_service
|
|
│ │ ├── ssh/ # SSH authorized_keys (id_ed25519.pub)
|
|
│ │ ├── etc/doas.d/ # doas config (4server.conf)
|
|
│ │ └── .profile # Shell profile for 4server user
|
|
│ │
|
|
│ ├── etc/traefik/certs/ # TLS certificates distributed to servers
|
|
│ ├── vault/ # Encrypted secrets (not committed)
|
|
│ │
|
|
│ ├── onboarding # New server bootstrap (Alpine packages, Nebula VPN, 4server user)
|
|
│ ├── hardening # Lock down SSH (no root, no password login)
|
|
│ ├── update # Full server update (packages, Docker infra, API, Traefik)
|
|
│ ├── update_sbin # Deploy /app/sbin/ to servers + restart services
|
|
│ ├── update_alpine # APK update/upgrade only
|
|
│ ├── restart # Restart core services
|
|
│ ├── backup_prune # Prune old Borg archives
|
|
│ ├── migrate # One-off DB migrations
|
|
│ └── download_sbin # Download binary tools (nebula, etc.)
|
|
│
|
|
├── exchange/ # Shared volume between host and control container (file drop)
|
|
├── docker-compose.yaml # Builds + runs the Alpine control container
|
|
├── mount # Helper to mount exchange volume
|
|
└── start # Start the control container
|
|
```
|
|
|
|
---
|
|
|
|
## Servers (Managed Fleet)
|
|
|
|
| Alias | IP | Port | Notes |
|
|
|------------|-----------------|------|--------------------|
|
|
| dev | 192.168.9.221 | 2222 | Development server |
|
|
| manchester | 192.168.9.20 | 22 | |
|
|
| boston | 192.168.9.16 | 22 | |
|
|
| mumbai | 192.168.9.17 | 22 | |
|
|
| meppel | 192.168.9.21 | 22 | |
|
|
| sydney | 192.168.9.22 | 22 | |
|
|
| saopaulo | 192.168.9.11 | 22 | |
|
|
|
|
SSH user on all servers: `4server`
|
|
Key per host: `/app/host_vars/{host}/{host}` (Ed25519)
|
|
Active hosts file: `/app/host_vars/hosts` (currently: `sydney2`)
|
|
|
|
---
|
|
|
|
## UUID Format
|
|
|
|
Every managed container has a UUID structured as: `{server}-{type}-{random}`
|
|
|
|
| Segment | Meaning | Known values |
|
|
|---------|-------------|-------------------------------------------|
|
|
| `aaa` | Server | `001` = manchester, `002` = boston |
|
|
| `bbb` | Image type | `001` = N8N, `002` = Odoo 18, `003` = Odoo 19, `004` = Odoo 17 |
|
|
|
|
The second segment (`bbb`) is used by `startContainer`, `nukeContainer`, `backupContainer` to dispatch to the right type-specific script.
|
|
|
|
---
|
|
|
|
## Core Tools (inside the Alpine control container)
|
|
|
|
### `rex <command>`
|
|
Runs a shell command on **all hosts** in `/app/host_vars/hosts` via `pssh`.
|
|
Example: `rex doas apk update`
|
|
|
|
### `prsync -h /app/host_vars/hosts -avz <src> <dst>`
|
|
Parallel rsync to all hosts. Used to deploy `/app/sbin/` and certs.
|
|
|
|
### `template <localfile> <remotefile>`
|
|
Renders `{{KEY}}` placeholders in `localfile` and copies it to `remotefile` on every host.
|
|
Available template variables (loaded from `host_vars/{host}/{host}.env` + key files):
|
|
- `NEBULA_CA`, `NEBULA_CRT`, `NEBULA_KEY`
|
|
- `API_KEY`, `HOSTNAME`
|
|
- `SSH_PRIVATE`, `SSH_PUBLIC`
|
|
|
|
---
|
|
|
|
## Server-Side Architecture (on each managed server)
|
|
|
|
```
|
|
/4server/
|
|
├── sbin/ # All management scripts (deployed from app/sbin/)
|
|
├── data/
|
|
│ ├── contracts.db # SQLite: container contracts table
|
|
│ ├── log/ # Logs for all services
|
|
│ ├── {UUID}/ # Per-container data directory
|
|
│ │ ├── odoo/ # Odoo filestore
|
|
│ │ ├── git/ # Git repos
|
|
│ │ ├── git-server/ # Bare git repos exposed via SSH
|
|
│ │ ├── logs/
|
|
│ │ ├── config/
|
|
│ │ ├── cc/ # Control channel (e.g. backup trigger file)
|
|
│ │ ├── etc/ # domain file, gitpath, etc.
|
|
│ │ ├── install/
|
|
│ │ └── .ssh/ # Container SSH keys
|
|
│ ├── traefik/etc/ # traefik.yaml + TLS certs
|
|
│ └── postgres/ # PostgreSQL data + pg_hba.conf
|
|
├── tmp/ # Temporary files (cleaned by cleanTmp service)
|
|
└── docker-compose.yml # Base infra (beedb + traefik)
|
|
|
|
/BACKUP/{UUID}/ # Per-container backup files (borgpush picks up current*)
|
|
```
|
|
|
|
### Persistent Docker Services (base infra)
|
|
| Container | Image | Role | IP (internal) |
|
|
|-----------|---------------|-------------------------------|----------------|
|
|
| `beedb` | postgres:16 | Shared PostgreSQL for all Odoo| 10.5.0.200 |
|
|
| `traefik` | traefik:3.6 | Reverse proxy, HTTPS (Let's Encrypt) | — |
|
|
|
|
Docker network: `4server_4projects` (bridge, `10.5.0.0/16`)
|
|
|
|
### Per-Container Docker Services
|
|
- Each UUID is a `docker run` container on the host's Docker network
|
|
- SSH port: `CONTAINERDBID + 2200` (forwarded to the host)
|
|
- Traefik routing via Docker labels (`traefik.http.routers.{UUID}.*`)
|
|
- Default domain: `{UUID}.odoo4projects.com`; optional custom domains from `/4server/data/{UUID}/etc/domain`
|
|
|
|
---
|
|
|
|
## Data Model: `contracts.db`
|
|
|
|
Table: `containers`
|
|
|
|
| Column | Description |
|
|
|----------------|----------------------------------------------------|
|
|
| `UUID` | Container identifier (aaa-bbb-... format) |
|
|
| `id` | Auto-increment row ID (`CONTAINERDBID`) |
|
|
| `image` | Image type code |
|
|
| `status` | Container status |
|
|
| `expires` | Contract expiry date |
|
|
| `created` | Creation timestamp |
|
|
| `secret_list` | Backslash-separated `key=value` pairs (passwords) |
|
|
| `git` | Git integration config |
|
|
| `backup_slots` | Number of backup slots |
|
|
| `hdd` | Disk quota |
|
|
| `workers` | Worker count |
|
|
| `domains_slots`| Number of allowed custom domains |
|
|
| `domains` | Active custom domains (comma-separated) |
|
|
| `affiliate` | JSON: UTM tracking data |
|
|
|
|
`secret_list` parsing: `key=value` pairs separated by `\`. Key `psql` = Odoo DB password (`ODOO_DB_PASSWORD`).
|
|
|
|
---
|
|
|
|
## Management API (`/4server/sbin/api`)
|
|
|
|
FastAPI Python 3 app, served by uvicorn, managed by OpenRC (`api` service).
|
|
|
|
- Auth: `X-API-Key` header (value from `/etc/4server` env file, key `API_KEY`)
|
|
- Version: `0.0.9`
|
|
|
|
Key endpoints (all protected):
|
|
| Endpoint | Description |
|
|
|----------|-------------|
|
|
| `GET /containers` | List all containers from SQLite |
|
|
| `POST /container/update` | Create/update a container record |
|
|
| `POST /container/start` | Start a container by UUID |
|
|
| `POST /container/stop` | Stop a container by UUID |
|
|
| `POST /container/nuke` | Destroy a container |
|
|
| `GET /container/info` | Full container info + Docker inspect |
|
|
| `POST /container/bump` | Bump/restart a container |
|
|
| `GET /container/quota` | Disk quota usage |
|
|
| `GET /images` | List available Docker images |
|
|
| `GET /cpu` | CPU log |
|
|
| `GET /stats` | Server runtime stats (JSON) |
|
|
| `GET /audit` | Full Docker + OS audit (JSON) |
|
|
| `POST /git` | Git pull/revert inside container |
|
|
| `POST /git/key` | Add SSH key to git-server |
|
|
| `GET /odoo/audit` | Odoo-specific audit |
|
|
| `GET /odoo/log` | Summarised Odoo log (last N lines, errors/warnings) |
|
|
| `POST /backup/import` | Import a backup archive |
|
|
| `POST /backup/move` | Move a backup file |
|
|
| `POST /backup/check` | Check backup integrity |
|
|
| `POST /backup/all` | Trigger backup of all containers |
|
|
| `POST /backup/borgpush` | Push backups to BorgBase |
|
|
|
|
---
|
|
|
|
## Backup
|
|
|
|
- **Per-container**: `backupContainer <UUID>` → dispatches to `backup/N8N` or `backup/ODOO`
|
|
- **All containers**: `backupAll` — touches `/4server/data/{UUID}/cc/backup` for each running NNN-NNN container
|
|
- **Off-site**: `borgpush` → BorgBase (`e7e9h45y.repo.borgbase.com`) via SSH key `~/.ssh/borg-backup`
|
|
- **Pruning**: `backup_prune` — keep 7 daily / 8 weekly / 12 monthly
|
|
|
|
---
|
|
|
|
## Security Model
|
|
|
|
- **Privilege escalation**: `doas` (not sudo). Config at `/etc/doas.d/4server.conf`
|
|
- **SSH**: Key-only (`PasswordAuthentication no`, `PermitRootLogin no`)
|
|
- **API**: Header-based API key (`X-API-Key`)
|
|
- **Secrets**: Stored in `secret_list` column of SQLite (base64-safe `\`-separated pairs)
|
|
- **VPN**: Nebula overlay mesh between all servers (certs per host in `host_vars/{host}/`)
|
|
- **TLS**: Traefik with Let's Encrypt production resolver
|
|
|
|
---
|
|
|
|
## Workflow Reference
|
|
|
|
### Add a new server
|
|
1. Add SSH key pair: `cd app/host_vars && ./create {hostname}`
|
|
2. Create `host_vars/{host}/{host}.env` with `HOSTNAME`, `API_KEY`, `NEBULA_CRT`, `NEBULA_KEY`
|
|
3. Add host to `host_vars/hosts`
|
|
4. Add SSH config block to `alpine/config`
|
|
5. Run `onboarding` from inside the control container
|
|
6. Run `hardening` to lock down SSH
|
|
|
|
### Deploy code changes to servers
|
|
```bash
|
|
# Inside control container (/app)
|
|
./update_sbin # deploy sbin/ scripts + restart api/checkCalls
|
|
```
|
|
|
|
### Deploy a config template
|
|
```bash
|
|
template templates/my.conf /etc/my.conf
|
|
```
|
|
|
|
### Run a command on all servers
|
|
```bash
|
|
rex doas rc-service api restart
|
|
```
|
|
|
|
### Start/stop a specific container (from server)
|
|
```bash
|
|
startContainer <UUID>
|
|
stopContainer <UUID>
|
|
```
|
|
|
|
---
|
|
|
|
## Key Technologies
|
|
|
|
| Layer | Technology |
|
|
|-------|------------|
|
|
| Control container | Alpine Linux + Docker Compose |
|
|
| Server OS | Alpine Linux (OpenRC init system) |
|
|
| Reverse proxy | Traefik v3.6 |
|
|
| Database | PostgreSQL 16 (shared, per-server) |
|
|
| App runtime | Odoo 17/18/19, N8N |
|
|
| Management API | Python 3 / FastAPI / uvicorn |
|
|
| Backup | BorgBackup → BorgBase |
|
|
| VPN | Nebula mesh |
|
|
| Parallel SSH | pssh / prsync |
|
|
| Container secrets | SQLite (`contracts.db`) |
|
|
| SSH auth | Ed25519 keys per host |
|