Watching

This page is optional. Your backend runs without it, and the step-by-step guide is complete without it. What it adds is a message on your phone when something breaks, instead of finding out from a user: the API stops answering, the certificate is about to expire, the nightly backup stopped running, or the disk is filling up.

The watcher is Uptime Kuma, run on the backend's own server the way Floot runs it: a compose project of its own beside Caddy, its web UI on the server's loopback, reached through the SSH tunnel, and its alerts sent to a Telegram chat. The page follows the guide's layout and names: api.myapp.com, you on your-server, the stack in ~/supabase-project, Caddy in ~/caddy, and the backup of step 17.

It watches two ways:

• HTTP monitors call your public endpoints every minute, by their public hostname, and check the answer and the certificate.
• Push monitors wait for the server to call them. The backup calls its monitor at the end of every run and a cron line calls the disk's every 5 minutes; a monitor that hears nothing by its deadline goes down, so a backup that stops, or a check that stops running, alerts too.

Kuma 2.x has no monitor export and no monitors as code: you create each monitor by hand, from the settings on this page, and this page is what you recreate them from.

1. Kuma beside Caddy

Kuma runs as its own compose project in ~/watcher, like Caddy in ~/caddy, so it starts, stops and upgrades on its own. Its one port is bound to the server's loopback: nothing on the internet can reach the UI, and the server's own pushes reach it at http://127.0.0.1:3001.

mkdir -p ~/watcher
nano ~/watcher/compose.yml

# Uptime Kuma, the watcher: its own compose project, beside Caddy's and the stack's.
#
# Its one port is published on the server's loopback, so the UI is reached only through the SSH
# tunnel, and the server's own pushes reach it at http://127.0.0.1:3001. It joins no network of
# the stack or of Caddy. The project name is pinned, so the volume (watcher_kuma_data: the
# monitors and their history) keeps its name. The image is an exact version, rootless and slim;
# UPTIME_KUMA_DB_TYPE picks SQLite, so the first visit asks only for the admin account.
name: watcher

services:
  kuma:
    image: louislam/uptime-kuma:2.5.5-slim-rootless
    restart: unless-stopped
    ports:
      - "127.0.0.1:3001:3001"
    environment:
      UPTIME_KUMA_DB_TYPE: sqlite
    volumes:
      - kuma_data:/app/data

volumes:
  kuma_data:

docker compose -f ~/watcher/compose.yml up -d

Check. Kuma listens on 127.0.0.1:3001 alone, and answers there:

sudo ss -tlnp | grep ':3001 '
curl -s http://127.0.0.1:3001/api/entry-page

ss prints one line, 127.0.0.1:3001, and curl prints {"type":"entryPage","entryPage":null}. From your laptop, curl -sS --max-time 5 http://your-server:3001/ fails. The loopback bind is what keeps it in: Docker bypasses ufw, so the firewall would not.

2. The UI, through the tunnel

The tunnel of step 12 gains a third forward, Kuma's port on 13001, clear of your local Supabase stack:

ssh -N -L 127.0.0.1:18000:127.0.0.1:8000 -L 127.0.0.1:15432:127.0.0.1:5433 \
  -L 127.0.0.1:13001:127.0.0.1:3001 you@your-server

While it runs, open http://localhost:13001. Do these two now, before anything else:

1. The admin account. The first page asks for one: a username and a new password, saved in your password manager. Until it exists, anyone with a shell on the server could claim it.
2. The base URL. The profile menu at the top right > Settings > General > Primary Base URL: http://127.0.0.1:3001, Save. Each push monitor shows its URL from this, and that is the address the server calls. Without it the URLs read localhost:13001, your end of the tunnel, which the server cannot reach.

Check. You are signed in to Kuma's dashboard, and Settings > General shows the base URL.

3. Telegram

Kuma sends its alerts through a Telegram bot of your own, to a chat only you are in:

1. In Telegram, open a chat with @BotFather, send /newbot, and give the bot a name (MyApp alerts) and a username ending in bot. BotFather answers with the bot's token, 123456789:AA...: save it in your password manager.

2. New Group, add the bot by its username, name the group (MyApp alerts), Create. In the group, send /start@<the bot's username>: a bot in a group reads only commands, and the next step needs one message it can read.

3. In Kuma: the profile menu > Settings > Notifications > Set Up Notification:

   Setting	Value
   Notification Type	Telegram
   Friendly Name	Telegram
   Bot Token	the token from BotFather
   Chat ID	Auto Get, shown once the token is in; check it is the group's (a group's starts with -)
   Use custom message template	off, so an alert reads [disk] [🔴 Down] <why>
   Default enabled	on, so every monitor you create gets it

   Press Test, then Save. Create the notification before the monitors, so each one gets it ticked.

Check. The test message arrived in the group, from your bot.

4. The HTTP monitors

Every monitor on this page alerts once when it goes down and once when it comes back, and stays quiet in between, which is what Resend Notification if Down X times consecutively at 0 means. Add New Monitor for each, with these settings in common:

and per monitor:

• api goes through Caddy, the gateway and Auth, the path every client takes. Auth serves its public keys to anyone, so the check needs no key; /auth/v1/health would answer 401, since the gateway asks every other Auth path for one.
• functions is the check of step 15: called with no token, the function itself answers 401 with the code NOT_AUTHENTICATED, before it reads anything. The keyword proves the function answered, not the gateway. The kit ships no health function of its own; when the functions fail to boot, on a missing or malformed secret, the answer is a 500 and the monitor goes down.
• The certificate. Caddy renews a certificate when about a third of its life is left, some 30 days for a 90-day certificate, so a warning below that means renewals are failing. Kuma warns once per threshold, at the days set under Settings > Notifications > TLS Certificate Expiry: 7, 14 and 21 by default. Remove 21, and keep every threshold under a third of your certificates' life, or a healthy certificate waiting for its renewal alerts. Caddy's log says why a renewal failed: docker compose -f ~/caddy/compose.yml logs caddy | grep -i error.

Kuma calls your public hostnames from the server itself, out and back in, as the functions already do: it proves DNS, Caddy, the certificate and the stack, but not that the internet can reach the server (see What it cannot see).

Check. Each monitor turns green within a minute, and api's page in Kuma shows the certificate's days left. Step 6 trips a monitor on purpose, to prove the alerts reach Telegram.

5. The backup's push monitor

The backup of step 17 runs every night. At the end of each run it calls its monitor: up when the run succeeded, down with the reason when it failed, so a failed run alerts at once. And when 25 hours pass with no call at all, a timer that stopped or a server that was off, the monitor goes down by itself. Add New Monitor:

Before Save, copy the Push URL the form shows, without its query: from http://127.0.0.1:3001/api/push/AbC123?status=up&msg=OK&ping=, keep http://127.0.0.1:3001/api/push/AbC123. Never use Reset Token on a monitor in use: the old URL stops working at once, and the next call to it is refused.

On the server, a drop-in adds the call to myapp-backup.service and leaves the backup script as it is. Save it as /etc/systemd/system/myapp-backup.service.d/watch.conf, with your push URL:

sudo install -d /etc/systemd/system/myapp-backup.service.d
sudo nano /etc/systemd/system/myapp-backup.service.d/watch.conf

# Tells Uptime Kuma's "backup" push monitor how each backup ended: up when it succeeded, down with
# systemd's reason when it failed, so a failed run alerts at once and a run that never comes trips
# the deadline. $$ passes a $ through systemd to the shell. The leading "-" keeps a failed push
# from changing the backup's own result; --retry-connrefused retries a Kuma that is restarting.
[Service]
ExecStopPost=-/bin/sh -c 'if [ "$$SERVICE_RESULT" = success ]; then s=up; else s=down; fi; \
  exec curl -fsS -m 10 --retry 3 --retry-connrefused -o /dev/null -G \
  --data-urlencode "status=$$s" \
  --data-urlencode "msg=backup $$SERVICE_RESULT, exit status $$EXIT_STATUS" \
  http://127.0.0.1:3001/api/push/AbC123'

A failed run then reads [backup] [🔴 Down] backup exit-code, exit status 1 in Telegram, and the backup's journal says why. Then run one backup, so the monitor hears its first call:

sudo systemctl daemon-reload
sudo systemctl start myapp-backup.service

Check. systemctl cat myapp-backup.service ends with the drop-in, the backup's journal shows backup <time> uploaded to ... as in step 17, and backup turns green in Kuma with the message backup success, exit status 0.

6. The disk check

A full disk stops Postgres, and Docker's images and volumes grow quietly. One cron line pushes the root filesystem's use to a monitor every 5 minutes: up under the limit, down over it, with the figure as the reason either way. Add New Monitor:

Copy its Push URL without the query, as above, and save the line as /etc/cron.d/myapp-watch, with your URL and, in l=80, the limit you want:

sudo nano /etc/cron.d/myapp-watch
sudo chmod 600 /etc/cron.d/myapp-watch

# Pushes the root filesystem's use to Uptime Kuma's "disk" push monitor every 5 minutes: up under
# the limit, down over it, the figure in the message either way. Cron reads a bare % as a new
# line, so each one here is written \%.
*/5 * * * * root l=80; u=$(df --output=pcent / | tr -dc 0-9); [ "$u" -lt "$l" ] && s=up || s=down; curl -fsS -m 10 -o /dev/null -G --data-urlencode "status=$s" --data-urlencode "msg=/ is $u\% used, the limit is $l\%" http://127.0.0.1:3001/api/push/AbC123

If df gives no number, the test fails and the line pushes down: a broken check never reads as a healthy disk.

Check. Within 5 minutes, disk turns green with a message such as / is 23% used, the limit is 80%. Then trip it once: set l=1 in the file, and at the next run Telegram gets [disk] [🔴 Down] / is 23% used, the limit is 1%. Set it back to 80; the next run sends the recovery.

When it goes down for real, docker system df shows how much Docker's images, containers and volumes hold, and sudo du -sh /var/lib/docker/containers the containers' logs, which step 7's rotation keeps at 30 MB per container.

A maintenance window, if you reboot on a schedule

The guide reboots by hand, in a quiet hour of your choosing, so an alert or two during a reboot tells you nothing new. If you reboot on a schedule instead, for example with unattended-upgrades' Automatic-Reboot "true" and an Automatic-Reboot-Time (a change from step 5 of the guide, whose check expects no automatic reboot), give Kuma a window over it, so the nightly reboot does not alert. Keep it clear of the backup's 03:30 UTC. The profile menu > Maintenance > Schedule Maintenance:

Leave backup out: its 25-hour deadline already covers a reboot. The form fills Effective Date Range with now to an hour later, its default for a single window, and keeps it when the strategy changes. Clear it: Kuma treats a window whose end date has passed as ended, and never holds an alert again.

Check. The maintenance list shows Reboot window as Scheduled, or Under Maintenance inside its hour, never Ended. During the window Kuma checks nothing and sends nothing; a monitor still down when it ends alerts then.

What it cannot see

A watcher on the server it watches cannot report that server down. When the server, Docker or the Kuma container stops, or Telegram is unreachable, no message comes: silence looks the same as all being well. And since Kuma calls your hostnames from inside the server, a fault between the internet and the server, such as a provider's firewall or a routing problem, is invisible to it too.

Floot accepts that silence. The fix, when you want one, is a single HTTP check from outside: a free external uptime service (UptimeRobot and Better Stack both have free plans) or a Kuma on another machine, calling https://api.myapp.com/auth/v1/.well-known/jwks.json every few minutes and alerting you by email. It is optional, and nothing on the server changes for it.

Upgrades

The image is pinned, so Kuma changes only when you change it: edit the tag in ~/watcher/compose.yml to a newer release, read its release notes first, and docker compose -f ~/watcher/compose.yml up -d. The volume keeps the monitors.

Floot's setup

Floot runs the same Kuma, 2.5.5-slim-rootless as the compose project watcher on 127.0.0.1:3001, reached through its SSH tunnel, with its alerts to Telegram only, one chat per server. Where it differs:

• Its HTTP monitors go through Cloudflare, for the docs, the apex's redirect, the web app's /version.json and the API's own health function, which answers 200 with the release's version, checked as a keyword, so a release needs no monitor edit.
• Its backup runs every 4 hours, so its push monitor's deadline is 4 hours 30 minutes.
• A recurring window, 03:30 to 04:30 UTC, over every monitor: Floot's server reboots there for pending updates.
• Six Host checks in place of the disk line: a root timer runs a script every 5 minutes, each check pushing to its own monitor, with a 15-minute deadline. They are listed for reference; the kit does not ship them.

CPU load and Docker itself are not checked: CPU alerts are noisy with nothing to act on, and Kuma stops with Docker. Floot's blind spot is the one above, and its named fix is the same external check, not built until customers justify it.