pywkt

Setting up a Local Build Server for Coolify on Proxmox

We've already got Coolify running on an LXC and things are looking good. Now let's set up another LXC in Proxmox to act as our build server. Because why wouldn't we?

This is a continuation of this post about setting up Coolify on an LXC and using Tailscale SSH to connect everything

Build Server

Create a new LXC in Proxmox

General:
- Name: coolify-build
- Leave as "Unprivileged"
Template: 
- Debian 12
Disks: 
- 60GB
CPU: 
- 4 Cores
Memory: 
- 8GB (8192MiB)
Network: 
- DHCP
DNS: 
- Default

Before starting the LXC we need to log in to the root node and add the following to the config

nano /etc/pve/lxc/<lxc-number>.conf

Add these lines to the bottom

lxc.cgroup.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

Save and exit.

Start the new coolify-build container

Update the system

apt update && apt upgrade

Install Curl

apt install curl

Tailscale

Log in to Tailscale and get a new script for setting up Tailscale on a linux server

Paste the script in the new Build LXC

Start Tailscale with SSH

tailscale up --ssh

If you followed the previous guide: Go to the Machines section in the Tailscale dashboard and add the ACL Tag coolify to the coolify-build node

Build LXC

Create a directory on the build server to hold all the registry data

cd ~ && mkdir registry-data && cd registry-data

Create a docker-compose.yml file in the registry directory

touch docker-compose.yml

Add the following to the docker-compose.yml

version: '3'
 
services:
  registry:
    image: registry:2
    ports:
      - "5000:5000"
    volumes:
      - ./registry-data:/var/lib/registry
      - ./auth:/auth
    environment:
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
    restart: unless-stopped

Save the docker-compose.yml file and create an auth directory in the registry directory

mkdir auth

Still in the registry directory, generate auth credentials

docker run --entrypoint htpasswd httpd:2 -Bbn <registry-username> <registry-password> > auth/htpasswd

Change <registry-username> and <registry-password> to unique values. We'll use these to log in later.

Change the permissions for the newly created htpasswd file

chmod 644 auth/htpasswd

Save the file and start the container. Whenever I run a new container for the first time I like to run it without the -d flag just so I can see what's happening and then once everything is working how I want it to I just stop the container and re-run it with the flag. So start the container

docker compose up

Note: Docker will automatically try to use HTTPS for connecting to the registry, but since all of this is staying on the local network I'm not going to bother with that which means we have to make a change to the server's Docker config.

If you set up HTTPS you can skip the following step.

Still on the Build LXC:

Stop the Docker container

docker compose down

Open the daemon.json file

nano /etc/docker/daemon.json

Add the following

{
  "insecure-registries": ["<build-server-tailscale-ip>:5000"], // Add this line
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
...rest of the config
}

Save the file and restart Docker

systemctl restart docker

Open a terminal on the Coolify LXC and make sure you're able to connect to the build server

ping <build-server-tailscale-ip> -p 5000

If the ping fails, double check your Tailscale ACL file and/or the UFW rules on the Coolify LXC

Once you're able to connect, log in to the Docker registry

docker login <build-server-tailscale-ip>:5000

Enter the username/password you created earlier

Login Succeeded

Now SSH in to the build server and log in again

ssh root@<build-server-tailscale-ip>
docker login <build-server-tailscale-ip>:5000

You will probably need to edit the /etc/docker/daemon.json file on the build server and the VPS. Just repeat the steps above for each server.

Note: I'm not sure why this file needs to be edited on all the servers, but when I remove it my builds fail 🤷

Make sure to run systemctl restart docker after each change to daemon.json

Coolify LXC

Connect to the Coolify LXC and make sure you can SSH to the Build LXC

ssh root@<build-lxc-tailscale-ip>

If everything's good, go back in the Coolify dashboard and set up the build server

Coolify dashboard

  • Click the Servers tab on the left menu
  • Click the + Add button at the top
  • Give the build server a name: Build LXC or whatever you want
  • Put in the Tailnet IP Address for the Build LXC
  • Keep the Private Key set to localhost's key
  • Check the box that says Use it as a build server?
  • Click Continue

Once you click the Verify button it should start to install a bunch of Docker stuff. Let that finish and we'll try to build something!

If you followed the last post and still have the blog template you can select that, otherwise add a new project.

Check the box that says Use a Build Server?

In the Docker Image textfield add

<build-server-tailscale-ip>:5000/<app-name>

Make sure your Build LXC docker compose is running if you stopped it earlier.

On the Build LXC

cd ~/registry-data
docker compose up -d

Save your settings, click the Deploy or Redeploy button at the top and watch your build-server CPU dance

If everything worked you should have a successful build and all the hard work was offloaded to another LXC. This lets us keep our Coolify LXC nice and lean.

Now let's set up our UFW rules on the Build LXC and call it a day

Build LXC

Install UFW

apt install ufw

Disable everything by default

ufw default deny incoming
ufw default deny outgoing

HTTPS and DNS are required to pull packages from the internet. This could probably be fine-tuned if you know exactly where all your packages are coming from, but for now this is fine.

Allow HTTPS

ufw allow in on 443
ufw allow out on 443

Allow out on HTTP

ufw allow out on 80/tcp

Allow DNS

ufw allow out on 53

Allow Tailscale

ufw allow in on tailscale0
ufw allow out on tailscale0

Stop and disable SSH

systemctl stop ssh
systemctl disable ssh
systemctl stop ssh.socket
systemctl disable ssh.socket

And there you have it! Coolify running locally on a Proxmox LXC with a local build server/Docker registry and everything communicates securley over an end-to-end encrypted tunnel thanks to Tailscale.

pywkt | code, open-source, privacy, self-hosting

gitlabgithub