> ## Documentation Index
> Fetch the complete documentation index at: https://docs.harly.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# Operations
> Health checks, the scheduler, backups, restore, and upgrades for a Harly installation.
## After reading this
* You will know what each health endpoint checks
* You will know how the scheduler's advisory locks prevent duplicate work
* You will know how to back up, restore, and verify a restore actually works
* You will know how to upgrade safely
## Health and logging
| Endpoint | Checks |
| ------------------- | ------------------------------------------------------------ |
| `/api/health/live` | Process only |
| `/api/health/ready` | PostgreSQL and the required migration, three-second deadline |
| `/api/health` | Alias for readiness |
All public probe responses expose only `status` and `version`. With PostgreSQL down, liveness stays `200` and readiness becomes `503`.
Every service rotates Docker JSON logs (10 MB x 3 files by default). The Next.js runtime cache is pruned at startup and every six hours, retaining at most 512 MB and seven days by default. Tune these without editing Compose:
```dotenv theme={null}
HARLY_LOG_MAX_SIZE=10m
HARLY_LOG_MAX_FILES=3
HARLY_CACHE_MAX_MB=512
HARLY_CACHE_MAX_AGE_DAYS=7
```
## Scheduler and queue runbook
The scheduler is a single, restartable worker. It runs the following jobs on
fixed intervals using PostgreSQL advisory locks — a second scheduler container
or restart receives `409 Already running` and skips rather than doing duplicate
work:
| Job | Interval |
| -------------------------- | ---------- |
| `domain-events` | 15 seconds |
| `email-outbox` | 1 minute |
| `webhooks-dispatch` | 1 minute |
| `esign-reconciliation` | 1 minute |
| `interview-sync` | 1 minute |
| `evaluation-jobs` | 1 minute |
| `document-expiry` | 1 minute |
| `retention-enforcement` | 1 minute |
| `candidate-reconciliation` | daily |
| `mail-reconciliation` | 1 minute |
| `scheduled-reports` | 1 minute |
| `mailbox-sync` | 2 minutes |
Queue rows left `processing` by a crash become eligible again after five minutes.
Mail deliveries left `sending` beyond the reconciliation lease become
`unknown`; they are never resent automatically because SMTP/IMAP may have
accepted the message before the process lost its response. Investigate the
provider outcome before resolving them.
The scheduler container's healthcheck runs its built-in doctor. It requires a recent successful-or-skipped record for **each** scheduled job, not merely a running container. The configured value is the minimum stale window; slower jobs also get a window of two scheduled intervals so daily reconciliation jobs are not falsely reported as stale:
```dotenv theme={null}
HARLY_SCHEDULER_STALE_AFTER_SECONDS=300
```
Change this only when the instance is intentionally suspended for longer than five minutes.
After a deployment, a restart, or an alert, run these from the installation directory:
```bash theme={null}
npx @harly/cli doctor
docker compose ps
docker compose logs --tail=100 scheduler
docker compose exec -T scheduler node /app/runtime.mjs doctor
```
The last command reports freshness per job plus email and webhook queue counts.
Do not manually invoke cron URLs or put `CRON_SECRET` in a URL. If a job is stale, first confirm app readiness, then restart only the scheduler with `docker compose restart scheduler`.
If queue counts grow while scheduler runs are fresh, fix the corresponding SMTP, webhook destination, or mailbox credentials; retries remain durable in PostgreSQL. Alert on an unhealthy scheduler, a stale job, and disk usage or backup failures — not merely on a non-zero queue count.
Uploads and PostgreSQL data are never pruned automatically. Monitor them with `docker system df -v` and keep encrypted backups outside the VPS.
The image exposes `serve`, `migrate`, `scheduler`, and `doctor`. Runtime migrations use committed SQL through `drizzle-orm`; Drizzle Kit and the TypeScript toolchain are build-time only.
## Backup, restore, and upgrade
```bash theme={null}
# Works everywhere: a private local rollback point (mode 0600).
npx @harly/cli backup
npx @harly/cli update --to --yes
npx @harly/cli restore backups/harly-2026-...tar.gz --force
# Optional advanced encryption for a portable backup.
AGE_RECIPIENT=age1... npx @harly/cli backup --encrypt
AGE_IDENTITY=/secure/key.txt npx @harly/cli restore backup.tar.gz.age --force
```
Backup stops app and scheduler, keeps PostgreSQL running, creates a `pg_dump -Fc`, includes local uploads and config, and checksums every included file before it can succeed. By default it creates a local rollback archive readable only by the deployment owner — this is convenient for a failed update, not a replacement for off-host disaster recovery.
Restore verifies every archived file checksum, creates a safety backup before changing data, stops the app, scheduler, and migration services to avoid a database race, restores PostgreSQL and local uploads, then requires readiness to succeed. A failed restore keeps the safety archive under `backups/`.
S3 objects are not embedded in the archive. Use bucket versioning and an off-provider backup policy alongside `npx @harly/cli backup`.
Upgrades take a backup first, run the single migrator, wait for readiness, and finish with `doctor`. Migrations are forward-only — restore the full backup when a release does not declare schema-compatible image rollback.
For managed platforms, use their managed PostgreSQL backups and snapshots together with S3 bucket encryption and versioning; they do not require installing `age`.
### Verify your backups actually restore
A backup you have never restored is a guess. Prove recovery end-to-end against a **throwaway** installation with the destructive harness:
```bash theme={null}
HARLY_DESTRUCTIVE_OK=1 \
tooling/harly/test/backup-restore.destructive.sh /path/to/installation
```
It seeds known rows and an uploaded object, backs them up, drops the entire database schema and deletes the upload, restores from the archive, and fails loudly unless row counts, a content checksum, the owner and user count, and the upload's SHA-256 all come back identical.
This harness refuses to run without `HARLY_DESTRUCTIVE_OK=1` because it wipes the database it points at. Never aim it at production.
## Development database
Repository contributors use a separate development Compose file, not the production stack:
```bash theme={null}
pnpm db:up
pnpm db:migrate
pnpm dev:web
```
It lives at `tooling/docker/compose.dev.yml`. Development needs Node 22+, pnpm, Docker, about 4 GB of available RAM, and 10 GB of free disk for a comfortable full-monorepo workflow. Turbopack's persistent filesystem cache is disabled to prevent multi-gigabyte `.next` growth, and its memory graph defaults to 1024 MB. Larger workstations can set `HARLY_DEV_MEMORY_MB=2048`; values below 512 MB are ignored. `pnpm clean` removes generated workspace caches without touching PostgreSQL volumes.
## Troubleshooting
### Scheduler doctor reports a stale job
**Symptom:** `docker compose exec -T scheduler node /app/runtime.mjs doctor` shows a job past its staleness window.
**Cause:** the scheduler container restarted, hung, or the instance was suspended longer than `HARLY_SCHEDULER_STALE_AFTER_SECONDS`.
**Solution:** confirm app readiness first, then `docker compose restart scheduler`. Do not invoke cron routes manually.
### Restore fails checksum verification
**Symptom:** `npx @harly/cli restore` exits before touching data.
**Cause:** the archive is corrupted or was modified after the backup ran.
**Solution:** use a different backup archive. The safety backup created before the attempted restore is preserved under `backups/`.
## Related pages
VPS requirements and DNS setup.
Choose Caddy, an external reverse proxy, or local-only.