Operations

For the person who owns the Linux host. Three commands bring the platform up; everything persistent lives in one directory you can back up with any file tool.

Requirements#

• a Linux host with Docker (and Compose v2);
• about 2 CPU and 4 GB RAM as a minimum — nested daemons are hungry;
• outbound Internet for image pulls: the host must forward and masquerade the hosting network so the platform can reach registries;
• ext4 (or another overlay-friendly filesystem) for data/.

Install in three commands#

git clone <this repo> && cd dind   # 1
make build                          # 2 — builds the management image (and MinIO); first time and after code changes
make up                             # 3 — idempotent bootstrap

make up is safe to run again at any time. It creates the hosting network, starts the main DinD, loads the platform images, creates the inner network, starts the management container and waits for it to report healthy. Then open http://10.10.10.10:8900 and sign in as admin@platform.example with the password printed in the management log (also in data/platform/secrets/panel.admin_password).

Day-2 commands#

Inside the main DinD, use docker exec dind-hosting docker … (for example docker exec dind-hosting docker ps).

Self-healing#

• Host reboot → Docker starts dind-hosting → every inner container has restart: unless-stopped → the stack returns without help.
• Management crash → restarted by its policy; on boot the reconciler re-asserts the network, the platform containers and every user's daemon (create if missing, start if stopped — never destroying user data).
• Daemon crash → its containers come back with it and the socket is recreated.
• A daemon that is merely still starting is given a grace window before it is considered broken.

Backing up data#

Everything persistent is under data/ next to the Makefile:

A consistent copy needs the platform stopped (make down), then a file-level copy of data/ (rsync, snapshot, tar), then make up. Keep data/platform/secrets with the rest — the encryption key for stored passwords lives there, and without it stored database passwords and secrets cannot be decrypted.

Upgrades#

git pull
make build            # new management image
scripts/upgrade.sh    # load it, recreate management, wait for health

Data is untouched; Alembic migrations run on boot. Roll back by rebuilding the previous commit and running scripts/upgrade.sh again.

• Runtime defaults (node, python, php, nginx versions) are in backend/app/services/deploy/runtimes.py, or override with PLATFORM_DEFAULT_*_VERSION — apps that follow the default move on their next rebuild.
• Database images for new databases are PLATFORM_DB_POSTGRES_IMAGE / PLATFORM_DB_MYSQL_IMAGE; existing databases keep their image. The platform's own database stays on postgres:16-alpine — do not point it at 17.

Production domains and TLS#

1. Point the domain's A record at the ingress in front of the host.
2. Make sure the ingress forwards port 80 and 443 to 10.10.10.10, and that the platform can reach the ACME CA.
3. Set the lb public ip(s) and a real contact email in platform → Settings / Certificates, then switch tls mode to prod.
4. Add domains; each must pass its pre-check first.

Real client addresses behind an L4 ingress#

If an nginx stream (or similar) fronts the platform, it can send the PROXY protocol so apps and logs see the real client address. It is off by default: set PLATFORM_TRAEFIK_PROXY_PROTOCOL_TRUSTED_IPS (or the per-entrypoint …_WEB_TRUSTED_IPS / …_WEBSECURE_TRUSTED_IPS) to the proxy's address. A trusted peer's header supplies the client address; an untrusted one is ignored; values that are malformed, /0 or overlap the platform network are rejected at boot.

Large uploads through the ingress#

A PUT of a 100+ MB archive to …/api/v1/uploads/<token> passes through your ingress: allow a request body of at least 512 MiB, disable request buffering for that path, and give it a generous timeout. Pass Host and X-Forwarded-Proto on the /mcp/ location, or set PLATFORM_PUBLIC_URL, so the upload link names the right address.

Disk usage#

Image layers and volumes accumulate inside each daemon. Check them per daemon:

docker exec dind-hosting docker system df
docker exec dind-hosting docker exec projects-<alias> docker system df

Prune build cache and dangling images inside the daemon with docker image prune / docker builder prune — never volume prune on a database daemon.

Container log limits#

Every container the platform creates — platform services, user daemons, apps, databases — keeps its log rotated at PLATFORM_LOG_MAX_SIZE (default 25m) × PLATFORM_LOG_MAX_FILE (default 2) = 50 MB at most; the newest 25 MB always survive a rotation. Set the two as CI/CD variables. Docker cannot change a log option on a running container, so a new value reaches management on the next rolling update, Traefik on the next management boot, and apps and databases on their next deploy or recreate; stateful platform containers keep their old setting until they are recreated.

Log locations#

Deploying the platform itself (CI)#

Production deploys run from GitLab CI on a dedicated runner (tag h0-infra): a status check, an update, and a manual deploy job. The pipeline calls the same scripts as above — nothing is changed over ssh. See the operator runbook (OPERATIONS.md) for the job list and the CI variables.