How to monitor Proxmox VE externally

The circular dependency problem

Ask a homelabber where their monitoring runs and a large share will answer: in a container on Proxmox. That is a monitoring system that can only report failures it survives, which excludes the one that matters most.

• A kernel update that does not come back after reboot. The node is off, every guest with it, and nothing is left to send mail.
• The root filesystem or local-lvm filling up, usually from backups or snapshots, which wedges the API before it wedges the guests.
• A failed power supply, which looks identical from outside to an ISP outage until you check both.
• Corosync losing quorum, so guests keep running but management stops responding and HA starts doing things you did not ask for.

An external check answers a question no internal tool can: is that machine reachable from the rest of the world right now.

What to check on a node

The Proxmox VE interface and API listen on TCP 8006 over HTTPS. Proxmox Backup Server is a separate daemon on 8007, also HTTPS, and deserves its own monitor. Both ship a self signed certificate, so an SSL check against a stock install fails chain validation until you install a real certificate.

The API lives under /api2/json. Watch /api2/json/version, which returns the running version, the point release, and the repository build id. Authenticate with an API token rather than a login ticket: tokens do not expire, need no CSRF token for GET requests, and can be scoped read only.

curl -fsS -k 'https://pve.example.com:8006/api2/json/version' \
  -H 'Authorization: PVEAPIToken=monitor@pve!undownable=SECRET-UUID-HERE'
# {"data":{"version":"...","release":"...","repoid":"..."}}

No Bearer, no quotes, the token id after an exclamation mark and the secret straight after the equals sign. Getting that syntax slightly wrong is the usual reason people give up. Put the whole string in as a custom request header, stored encrypted.

Scope the token before you use it

Create a dedicated user, grant the built in read only PVEAuditor role, then create a token. A trap catches almost everyone here: Proxmox API tokens have privilege separation on by default, so the token gets its own empty ACL and inherits nothing.

pveum user add monitor@pve
pveum acl modify / --users monitor@pve --roles PVEAuditor
pveum user token add monitor@pve undownable --privsep 0

A JSON check asserting data.version beats a bare liveness probe: it fails when the API answers with an error envelope, and it documents which version each node runs every time it passes.

Run the ping check alongside the API check and the pair tells you which layer failed. Ping green and API red means pveproxy or the certificate; both red means the machine or the network.

When the API is not reachable from outside

Publishing a hypervisor management API to the internet is not something to do casually. The alternative gives up nothing: have the node call out on a schedule and treat missing calls as the incident.

Put the cron entry on the node, not in a guest. A heartbeat from a VM only proves that VM is alive:

*/2 * * * * curl -fsS -m 10 https://undownable.com/ping/your-monitor-id > /dev/null

To make it reflect the API rather than just the kernel, gate it on a loopback probe where skipping certificate validation is safe:

*/2 * * * * curl -fsS -k -m 10 -o /dev/null \
  -H "Authorization: PVEAPIToken=$PVE_TOKEN" \
  https://127.0.0.1:8006/api2/json/version \
  && curl -fsS -m 10 https://undownable.com/ping/your-monitor-id > /dev/null

One heartbeat monitor per node, named after the node. A three node cluster is three monitors, comfortably inside the free plan, and when one goes quiet you know which machine to walk over to.

Apply the same pattern to backups: append the ping to the end of your vzdump or Proxmox Backup Server job script. A job that stops running is then reported that day, not the day you need a restore.

Alerting that does not depend on the node

Postfix on the node is a fine secondary path, not a primary one, because it needs the node up, the network up, and a relay reachable. External alerts need none of those:

• ntfy, including self hosted, using an access token beginning with tk_ sent as a bearer token. Host it somewhere other than the cluster you are watching.
• Telegram, the honest answer for most homelabs because it depends on nothing you own.
• Email as a backstop, and a webhook if you want to drive something else.

Use confirmation thresholds so patch reboots do not page you at 2am, and a maintenance window over upgrade night. Degraded state helps too: a node answering slowly often has a storage problem coming.