Setting up Prometheus and Grafana on Proxmox LXCs

What I'm doing:

Setting up three LXC containers in my Proxmox node.
- Prometheus
- prometheus-pve-exporter
- Grafana.
pve-exporter will pull data from the Proxmox server
Prometheus will pull the data from the pve-exporter container
Grafana will pull from the Prometheus container and display it nice and pretty.

I assume you're already wondering why I would use three separate containers when I could just do this on one or even two. There's a few reasons, but the main one is that I prefer to have a bunch of small containers each doing one thing rather than a few larger containers each doing several things. Personal preference, but it works for me 🤷‍♀️

The following instructions assume you already have a Proxmox server up and running and know how to create LXC containers.

All the containers created in these steps are created with the Ubuntu Server 22.04 image.

Setting up the Prometheus container

2cpu, 2gb ram, 8gb storage. Calling this one 'prometheus'

The steps in this section are taken directly from samynitsche.de

Create a group/user

groupadd --system prometheus \
useradd -s /sbin/nologin --system -g prometheus prometheus

Create directories

mkdir /var/lib/prometheus \
for i in rules rules.d files_sd; do mkdir -p /etc/prometheus/${i}; done

Download/install Prometheus

mkdir -p /tmp/prometheus && cd /tmp/prometheus \
curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest \
  | grep browser_download_url \
  | grep linux-amd64 \
  | cut -d '"' -f 4 \
  | wget -qi -

Extract files and move to correct directory

tar xvf prometheus*.tar.gz \
cd prometheus \
mv prometheus promtool /usr/local/bin/ \
mv prometheus.yml  /etc/prometheus/prometheus.yml \
mv consoles/ console_libraries/ /etc/prometheus/

Create systemd config

cd /etc/systemd/system/ && touch prometheus.service

nano prometheus.service

Paste in the following and save

[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target
 
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url=
 
SyslogIdentifier=prometheus
Restart=always
 
[Install]
WantedBy=multi-user.target

Set permissions

for i in rules rules.d files_sd; do chown -R prometheus:prometheus /etc/prometheus/${i}; done

for i in rules rules.d files_sd; do chmod -R 775 /etc/prometheus/${i}; done

chown -R prometheus:prometheus /var/lib/prometheus/

Start Prometheus

systemctl daemon-reload

systemctl start prometheus

systemctl enable prometheus

You should now be able to go to http://<container-ip>:9090 and see the Prometheus instance running

Installing proxmox-pve-exporter

The following steps are a modified version of this post.

Create a new container

1cpu, 1gb ram, 6gb storage. Calling it 'prom-export'

Install pip3

apt-get install python3-pip

pip3 install prometheus-pve-exporter

Make directory for pve_exporter

mkdir -p /usr/share/pve_exporter/

Add Proxmox credentials

nano /usr/share/pve_exporter/pve_exporter.yml

Add the following, but change to your Proxmox info

default:
    user: proxmox_pve_user@pve
    password: secret_password
    verify_ssl: false

Save and exit

Test that it works

/usr/local/bin/pve_exporter /usr/share/pve_exporter/pve_exporter.yml

After running that in the console you should see something like:

WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://:9221
Press CTRL+C to quit

Create systemd script

nano /etc/systemd/system/pve_exporter.service

Add the following to the file:

[Unit]
Description=Proxmox VE Prometheus Exporter
After=network.target
Wants=network.target
 
[Service]
Restart=on-failure
WorkingDirectory=/usr/share/pve_exporter
ExecStart=/usr/local/bin/pve_exporter /usr/share/pve_exporter/pve_exporter.yml 9221 192.168.X.X
 
[Install]
WantedBy=multi-user.target

Replace the IP address in that script with the IP of this container

Save and exit

Start pve_exporter service

systemctl daemon-reload

systemctl enable pve_exporter

systemctl start pve_exporter

Check that it's running with:

systemctl status pve_exporter

Back on the Prometheus container

Add the pve_exporter data to Prometheus

nano /etc/prometheus/prometheus.yml

Add another job_name after the line - targets: ["localhost:9000"]

  - job_name: 'proxmox'
    metrics_path: '/pve'
    params:
        # This is the ip address of the proxmox server
        target: ['192.168.X.X']
    static_configs:
      # This is the container running pve_exporter
      - targets: ["192.168.X.X:9221"]

Save and exit

Restart Prometheus

systemctl restart prometheus

You should now be able to go to http://<prometheus-container-ip>:9090/targets and see two entries. The first one is there by default, it's the Prometheus container. The second one should be the Proxmox metrics.

Setup Grafana

Make new container

2cpu, 2gb ram, 8gb storage. Calling it 'grafana'.

Setup apt for Grafana

apt-get install apt-transport-https software-properties-common wget

wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key

echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | tee -a /etc/apt/sources.list.d/grafana.list

Install Grafana

apt-get update

apt-get install grafana-enterprise

Start Grafana

systemctl daemon-reload

systemctl enable grafana-server.service

systemctl start grafana-server

Check that it's running

systemctl status grafana-server

If everything looks good, you should be able to go to http://<grafana-container-ip>:3000 and see the login screen

Configure Grafana with Prometheus

Once logged in to Grafana:

Click the gear icon at the bottom left
Click the "Add data source" button
Select Prometheus
In the URL field, add the Prometheus container's IP and port like: http://192.168.X.X:9090
Click "Save and test"

Set up the pve-exporter dashboard

Click the Dashboards icon on the left menu and select + Import
In the textfield that says "Import via grafana.com" enter 10347 and click "Load"
Select Prometheus as the data source and save

Now you should have all your Proxmox metrics in this Grafana dashboard.

There's a chance that some of the labels may be overlapping and look all messed up. If so, just click the dropdown arrow at the top of that section and click "Edit". Expand the "Options" accordion and change the textfield that says {{name}} to {{id}}.

Refs:

pywkt

Setting up Prometheus and Grafana on Proxmox LXCs

Setting up the Prometheus container

Create a group/user

Create directories

Download/install Prometheus

Extract files and move to correct directory

Create systemd config

Set permissions

Start Prometheus

Installing proxmox-pve-exporter

Create a new container

Install pip3

Make directory for pve_exporter

Add Proxmox credentials

Test that it works

Create systemd script

Start pve_exporter service

Back on the Prometheus container

Add the pve_exporter data to Prometheus

Restart Prometheus

Setup Grafana

Make new container

Setup apt for Grafana

Install Grafana

Start Grafana

Check that it's running

Configure Grafana with Prometheus

Set up the pve-exporter dashboard